From e5884a35398c58c6f19e3e40f93c25ee51ef77ef Mon Sep 17 00:00:00 2001
From: Eric Pugh <epugh@opensourceconnections.com>
Date: Tue, 29 Oct 2024 07:56:37 -0400
Subject: [PATCH 01/32] Track changes for the whole slew of CLI commands under
 one umbrella JIRA (#2810)

---
 solr/CHANGES.txt | 8 +++-----
 1 file changed, 3 insertions(+), 5 deletions(-)

diff --git a/solr/CHANGES.txt b/solr/CHANGES.txt
index 1ddf5f4be45..a4921ec4065 100644
--- a/solr/CHANGES.txt
+++ b/solr/CHANGES.txt
@@ -135,11 +135,9 @@ Improvements
   which may help reduce distributed-search latency in collections with many shards, especially
   when PKI is used between nodes. (Jason Gerlowski)
 
-* SOLR-17382: Deprecate -a and -addlopts in favour of --jvm-opts for passing options into the JVM in bin/solr.   (Eric Pugh, Christos Malliaridis)
-
-* SOLR-17431: Deprecate -p parameter where it doesn't refer to a port in bin/solr. (Eric Pugh, Christos Malliaridis)
-
-* SOLR-17442: Resolve -v flag conflict (version, value, verbose) in bin/solr. (Eric Pugh, Christos Malliaridis)
+* SOLR-17383: Resolved overlapping arguments in the Solr CLI.  Removed duplicative but differing arguments,
+  consolidated use of short form arguments -v to not have differing meanings based on tool.  Provide deprecation warning 
+  in command line when deprecated arguments are used. (Eric Pugh, Christos Malliaridis)
 
 * SOLR-17256: Deprecate SolrRequest `setBasePath` and `getBasePath` methods.  SolrJ users wishing to temporarily
   override an HTTP client's base URL may use `Http2SolrClient.requestWithBaseUrl` instead. (Jason Gerlowski,

From 42f8e485446c3bb82aa5d0cd0ef7196059ddc8fb Mon Sep 17 00:00:00 2001
From: Eric Pugh <epugh@opensourceconnections.com>
Date: Tue, 29 Oct 2024 09:49:52 -0400
Subject: [PATCH 02/32] SOLR-17528: Match PackageTool use of -y (#2813)

--no-prompt now has short version -y
---
 solr/CHANGES.txt                                           | 2 ++
 solr/core/src/java/org/apache/solr/cli/RunExampleTool.java | 2 +-
 2 files changed, 3 insertions(+), 1 deletion(-)

diff --git a/solr/CHANGES.txt b/solr/CHANGES.txt
index a4921ec4065..ac00cf68152 100644
--- a/solr/CHANGES.txt
+++ b/solr/CHANGES.txt
@@ -147,6 +147,8 @@ Improvements
   enqueuing with a risk of rejection. Solr will use less resources under stress but to get the most
   of your machine, you may want to increase the thread pool. (David Smiley)
 
+* SOLR-17528: Introduce -y short option to bin/solr start --no-prompt option. Aligns with bin/solr package tool.  (Eric Pugh)
+
 Optimizations
 ---------------------
 * SOLR-14985: Solrj CloudSolrClient with Solr URLs had serious performance regressions (since the
diff --git a/solr/core/src/java/org/apache/solr/cli/RunExampleTool.java b/solr/core/src/java/org/apache/solr/cli/RunExampleTool.java
index 788a9b86dfc..95c746f82b7 100644
--- a/solr/core/src/java/org/apache/solr/cli/RunExampleTool.java
+++ b/solr/core/src/java/org/apache/solr/cli/RunExampleTool.java
@@ -94,7 +94,7 @@ public String getName() {
   @Override
   public List<Option> getOptions() {
     return List.of(
-        Option.builder()
+        Option.builder("y")
             .longOpt("no-prompt")
             .required(false)
             .desc(

From 1e5d7599e16630ab2b676a6778a52ade7d3474fe Mon Sep 17 00:00:00 2001
From: Eric Pugh <epugh@opensourceconnections.com>
Date: Tue, 29 Oct 2024 10:05:47 -0400
Subject: [PATCH 03/32] SOLR-17384: Bring ExportTool options in line with the
 new CLI pattern (#2807)

Co-authored-by: Christos Malliaridis <c.malliaridis@gmail.com>
---
 .../java/org/apache/solr/cli/ExportTool.java  | 35 +++++++++++++++----
 .../org/apache/solr/cli/TestExportTool.java   |  4 +--
 solr/packaging/test/test_basic_auth.bats      |  2 +-
 solr/packaging/test/test_export.bats          | 18 +++++-----
 .../pages/solr-control-script-reference.adoc  | 28 ++++++++++-----
 5 files changed, 60 insertions(+), 27 deletions(-)

diff --git a/solr/core/src/java/org/apache/solr/cli/ExportTool.java b/solr/core/src/java/org/apache/solr/cli/ExportTool.java
index e2167c39091..97e4774f391 100644
--- a/solr/core/src/java/org/apache/solr/cli/ExportTool.java
+++ b/solr/core/src/java/org/apache/solr/cli/ExportTool.java
@@ -116,28 +116,48 @@ public List<Option> getOptions() {
             .desc("Name of the collection.")
             .build(),
         Option.builder("out")
+            .deprecated(
+                DeprecatedAttributes.builder()
+                    .setForRemoval(true)
+                    .setSince("9.8")
+                    .setDescription("Use --output instead")
+                    .get())
+            .hasArg()
+            .argName("PATH")
+            .desc(
+                "Path to output the exported data, and optionally the file name, defaults to 'collection-name'.")
+            .build(),
+        Option.builder()
+            .longOpt("output")
             .hasArg()
             .argName("PATH")
             .desc(
                 "Path to output the exported data, and optionally the file name, defaults to 'collection-name'.")
             .build(),
-        Option.builder("format")
+        Option.builder()
+            .longOpt("format")
             .hasArg()
             .argName("FORMAT")
             .desc("Output format for exported docs (json, jsonl or javabin), defaulting to json.")
             .build(),
-        Option.builder("compress").desc("Compress the output.").build(),
-        Option.builder("limit")
+        Option.builder()
+            .longOpt("compress")
+            .desc("Compress the output. Defaults to false.")
+            .build(),
+        Option.builder()
+            .longOpt("limit")
             .hasArg()
             .argName("#")
             .desc("Maximum number of docs to download. Default is 100, use -1 for all docs.")
             .build(),
-        Option.builder("query")
+        Option.builder()
+            .longOpt("query")
             .hasArg()
             .argName("QUERY")
             .desc("A custom query, default is '*:*'.")
             .build(),
-        Option.builder("fields")
+        Option.builder()
+            .longOpt("fields")
             .hasArg()
             .argName("FIELDA,FIELDB")
             .desc("Comma separated list of fields to export. By default all fields are fetched.")
@@ -280,8 +300,11 @@ public void runImpl(CommandLine cli) throws Exception {
     String credentials = cli.getOptionValue(SolrCLI.OPTION_CREDENTIALS.getLongOpt());
     Info info = new MultiThreadedRunner(url, credentials);
     info.query = cli.getOptionValue("query", "*:*");
+
     info.setOutFormat(
-        cli.getOptionValue("out"), cli.getOptionValue("format"), cli.hasOption("compress"));
+        SolrCLI.getOptionWithDeprecatedAndDefault(cli, "output", "out", null),
+        cli.getOptionValue("format"),
+        cli.hasOption("compress"));
     info.fields = cli.getOptionValue("fields");
     info.setLimit(cli.getOptionValue("limit", "100"));
     info.output = super.stdout;
diff --git a/solr/core/src/test/org/apache/solr/cli/TestExportTool.java b/solr/core/src/test/org/apache/solr/cli/TestExportTool.java
index 1749d7c54d9..6347c4a47e7 100644
--- a/solr/core/src/test/org/apache/solr/cli/TestExportTool.java
+++ b/solr/core/src/test/org/apache/solr/cli/TestExportTool.java
@@ -249,9 +249,9 @@ public void testWithBasicAuth() throws Exception {
         COLLECTION_NAME,
         "--credentials",
         SecurityJson.USER_PASS,
-        "-out",
+        "--output",
         outFile.getAbsolutePath(),
-        "-verbose"
+        "--verbose"
       };
 
       assertEquals(0, runTool(args));
diff --git a/solr/packaging/test/test_basic_auth.bats b/solr/packaging/test/test_basic_auth.bats
index e4f84aabd93..333d5cf73d3 100644
--- a/solr/packaging/test/test_basic_auth.bats
+++ b/solr/packaging/test/test_basic_auth.bats
@@ -77,7 +77,7 @@ run solr create -c COLL_NAME
   assert_output --partial 'Committed'
   
   # Test export
-  #run solr export -u name:password --solr-url "http://localhost:${SOLR_PORT} --name COLL_NAME" -query "*:*" -out "${BATS_TEST_TMPDIR}/output"
+  #run solr export -u name:password --solr-url http://localhost:${SOLR_PORT} --name COLL_NAME --query "*:*" --output "${BATS_TEST_TMPDIR}/output"
   #assert_output --partial 'Export complete'
   
 }
diff --git a/solr/packaging/test/test_export.bats b/solr/packaging/test/test_export.bats
index 0ac04ce3d02..5dc7c9833b6 100644
--- a/solr/packaging/test/test_export.bats
+++ b/solr/packaging/test/test_export.bats
@@ -30,32 +30,32 @@ teardown() {
 
 @test "Check export command" {
   run solr start -e techproducts
-  run solr export --solr-url http://localhost:${SOLR_PORT} --name techproducts -query "*:* -id:test" -out "${BATS_TEST_TMPDIR}/output"
+  run solr export --solr-url http://localhost:${SOLR_PORT} --name techproducts --query "*:* -id:test" --output "${BATS_TEST_TMPDIR}/output"
 
   refute_output --partial 'Unrecognized option'
   assert_output --partial 'Export complete'
 
   assert [ -e ${BATS_TEST_TMPDIR}/output.json ]
 
-  run solr export -url "http://localhost:${SOLR_PORT}/solr/techproducts" -query "*:* -id:test"
+  run solr export --solr-url http://localhost:${SOLR_PORT} -c techproducts --query "*:* -id:test"
   assert [ -e techproducts.json ]
   rm techproducts.json
 
-  run solr export -url "http://localhost:${SOLR_PORT}/solr/techproducts" -query "*:* -id:test" -format javabin
+  run solr export --solr-url http://localhost:${SOLR_PORT} -c techproducts --query "*:* -id:test" --format javabin
   assert [ -e techproducts.javabin ]
   rm techproducts.javabin
 
-  # old pattern of putting a suffix on the out that controlled the format no longer supported ;-).
-  run solr export -url "http://localhost:${SOLR_PORT}/solr/techproducts" -query "*:* -id:test" -out "${BATS_TEST_TMPDIR}/output.javabin"
+  # old pattern of putting a suffix on the output that controlled the format no longer supported ;-).
+  run solr export --solr-url http://localhost:${SOLR_PORT} -c techproducts --query "*:* -id:test" --output "${BATS_TEST_TMPDIR}/output.javabin"
   assert [ -e ${BATS_TEST_TMPDIR}/output.javabin.json ]
 
-  run solr export -url "http://localhost:${SOLR_PORT}/solr/techproducts" -query "*:* -id:test" -out "${BATS_TEST_TMPDIR}"
+  run solr export --solr-url http://localhost:${SOLR_PORT} -c techproducts --query "*:* -id:test" --output "${BATS_TEST_TMPDIR}"
   assert [ -e ${BATS_TEST_TMPDIR}/techproducts.json ]
 
-  run solr export -url "http://localhost:${SOLR_PORT}/solr/techproducts" -query "*:* -id:test" -format jsonl -out "${BATS_TEST_TMPDIR}/output"
+  run solr export --solr-url http://localhost:${SOLR_PORT} -c techproducts --query "*:* -id:test" --format jsonl --output "${BATS_TEST_TMPDIR}/output"
   assert [ -e ${BATS_TEST_TMPDIR}/output.jsonl ]
 
-  run solr export -url "http://localhost:${SOLR_PORT}/solr/techproducts" -query "*:* -id:test" -limit 10 -compress -format jsonl -out "${BATS_TEST_TMPDIR}/output"
+  run solr export --solr-url http://localhost:${SOLR_PORT} -c techproducts --query "*:* -id:test" --limit 10 --compress --format jsonl --output "${BATS_TEST_TMPDIR}/output"
   assert [ -e ${BATS_TEST_TMPDIR}/output.jsonl.gz ]
   assert_output --partial 'Total Docs exported: 10'
 
@@ -64,7 +64,7 @@ teardown() {
 @test "export fails on non cloud mode" {
   run solr start --user-managed
   run solr create -c COLL_NAME
-  run solr export -url "http://localhost:${SOLR_PORT}/solr/COLL_NAME"
+  run solr export --solr-url http://localhost:${SOLR_PORT} -c COLL_NAME
   refute_output --partial 'Export complete'
   assert_output --partial "ERROR: Couldn't initialize a HttpClusterStateProvider"
 }
diff --git a/solr/solr-ref-guide/modules/deployment-guide/pages/solr-control-script-reference.adoc b/solr/solr-ref-guide/modules/deployment-guide/pages/solr-control-script-reference.adoc
index 5d51d643a96..c0e18e46091 100644
--- a/solr/solr-ref-guide/modules/deployment-guide/pages/solr-control-script-reference.adoc
+++ b/solr/solr-ref-guide/modules/deployment-guide/pages/solr-control-script-reference.adoc
@@ -1634,16 +1634,27 @@ NOTE: The `export` command only works with in a Solr running in cloud mode.
 
 The `bin/solr export` command takes the following parameters:
 
-`--url <url>`::
+`-s <url>` or `--solr-url <url>`::
++
+[%autowidth,frame=none]
+|===
+|Optional |Default: `http://localhost:8983`
+|===
++
+Fully-qualified address to a Solr.
++
+*Example*: `--solr-url http://localhost:8983`
+
+`-c <collection>`::
 +
 [%autowidth,frame=none]
 |===
 s|Required |Default: none
 |===
 +
-Fully-qualified address to a collection.
+Name of the collection to run an export against.
 +
-*Example*: `--url http://localhost:8983/solr/techproducts`
+*Example*: `bin/solr export -c gettingstarted`
 
 `--format <format>`::
 +
@@ -1656,7 +1667,7 @@ The file format of the export, `json`, `jsonl`, or `javabin`.
 Choosing `javabin` exports in the native Solr format, and is compact and fast to import.
 `jsonl` is the Json with Lines format, learn more at https://jsonlines.org/.
 
-`--out <path>`::
+`--output <path>`::
 +
 [%autowidth,frame=none]
 |===
@@ -1696,7 +1707,6 @@ The default is `\*:*` which will export all documents.
 Comma separated list of fields to be exported.
 By default all fields are fetched.
 
-
 `--limit <number of documents>`::
 +
 [%autowidth,frame=none]
@@ -1722,13 +1732,13 @@ This parameter is unnecessary if `SOLR_AUTH_TYPE` is defined in `solr.in.sh` or
 Export all documents from a collection `gettingstarted`:
 
 [source,bash]
-bin/solr export --url http://localhost:8983/solr/gettingstarted --limit -1
+bin/solr export --solr-url http://localhost:8983 -c gettingstarted --limit -1
 
 Export all documents of collection `gettingstarted` into a file called `1MDocs.json.gz` as a compressed JSON file:
 
 [source,bash]
 ----
-bin/solr export --url http://localhost:8983/solr/gettingstarted --limit -1 --format json --compress --out 1MDocs
+bin/solr export --solr-url http://localhost:8983 -c gettingstarted --limit -1 --format json --compress --output 1MDocs
 ----
 
 === Importing Documents into a Collection
@@ -1741,7 +1751,7 @@ First export the documents, making sure to ignore any fields that are populated
 
 [,console]
 ----
-$ bin/solr export --url http://localhost:8983/solr/gettingstarted --fields id,name,manu,cat,features
+$ bin/solr export --solr-url http://localhost:8983 -c gettingstarted --fields id,name,manu,cat,features
 ----
 
 Create a new collection to import the exported documents into:
@@ -1767,7 +1777,7 @@ $ curl -H 'Content-Type: application/json' -X POST -d @gettingstarted.json 'http
 
 [,console]
 ----
-$ bin/solr export --url http://localhost:8983/solr/gettingstarted --format javabin --fields id,name,manu,cat,features
+$ bin/solr export --solr-url http://localhost:8983 -c gettingstarted --format javabin --fields id,name,manu,cat,features
 $ curl -X POST --header "Content-Type: application/javabin" --data-binary @gettingstarted.javabin 'http://localhost:8983/solr/test_collection/update?commit=true'
 ----
 

From dd6f2152c236d301f84e328b84e21ac89b2c9145 Mon Sep 17 00:00:00 2001
From: Eric Pugh <epugh@opensourceconnections.com>
Date: Tue, 29 Oct 2024 11:19:05 -0400
Subject: [PATCH 04/32] SOLR-17526: Deprecate --out in favour of --verbose.
 (#2812)

---
 solr/core/src/java/org/apache/solr/cli/PostTool.java | 11 ++++++++++-
 solr/packaging/test/test_post.bats                   | 12 ++++++++++++
 .../indexing-guide/pages/indexing-with-tika.adoc     |  2 +-
 .../modules/indexing-guide/pages/post-tool.adoc      | 11 +++++------
 4 files changed, 28 insertions(+), 8 deletions(-)

diff --git a/solr/core/src/java/org/apache/solr/cli/PostTool.java b/solr/core/src/java/org/apache/solr/cli/PostTool.java
index 059ae8a1def..d479a90aaba 100644
--- a/solr/core/src/java/org/apache/solr/cli/PostTool.java
+++ b/solr/core/src/java/org/apache/solr/cli/PostTool.java
@@ -278,6 +278,12 @@ public List<Option> getOptions() {
             .build(),
         Option.builder()
             .longOpt("out")
+            .deprecated(
+                DeprecatedAttributes.builder()
+                    .setForRemoval(true)
+                    .setSince("9.8")
+                    .setDescription("Use --verbose instead")
+                    .get())
             .required(false)
             .desc("sends Solr response outputs to console.")
             .build(),
@@ -360,7 +366,10 @@ public void runImpl(CommandLine cli) throws Exception {
 
     recursive = Integer.parseInt(cli.getOptionValue("recursive", "1"));
 
-    out = cli.hasOption("out") ? CLIO.getOutStream() : null;
+    out =
+        cli.hasOption("out") || cli.hasOption(SolrCLI.OPTION_VERBOSE.getLongOpt())
+            ? CLIO.getOutStream()
+            : null;
     commit = !(cli.hasOption("skipcommit") || cli.hasOption("skip-commit"));
     optimize = cli.hasOption("optimize");
 
diff --git a/solr/packaging/test/test_post.bats b/solr/packaging/test/test_post.bats
index 3d796820029..016dba34d36 100644
--- a/solr/packaging/test/test_post.bats
+++ b/solr/packaging/test/test_post.bats
@@ -203,3 +203,15 @@ capture_echo_to_solr() {
   run capture_echo_to_solr
   assert_output --partial '"status":0'
 }
+
+@test "verbose echo the Solr response" {
+
+  run solr create -c monitors_verbose -d _default
+  assert_output --partial "Created collection 'monitors_verbose'"
+
+  run solr post --verbose --type application/xml --solr-update-url http://localhost:${SOLR_PORT}/solr/monitors_verbose/update ${SOLR_TIP}/example/exampledocs/monitor.xml
+
+  assert_output --partial '1 files indexed.'
+  assert_output --partial '<lst name="responseHeader">'
+  refute_output --partial 'ERROR'
+}
diff --git a/solr/solr-ref-guide/modules/indexing-guide/pages/indexing-with-tika.adoc b/solr/solr-ref-guide/modules/indexing-guide/pages/indexing-with-tika.adoc
index 3d4e1509fbe..b1344ed5519 100644
--- a/solr/solr-ref-guide/modules/indexing-guide/pages/indexing-with-tika.adoc
+++ b/solr/solr-ref-guide/modules/indexing-guide/pages/indexing-with-tika.adoc
@@ -603,7 +603,7 @@ The output includes XML generated by Tika (and further escaped by Solr's XML) us
 
 [,console]
 ----
-$ bin/solr post -c gettingstarted --params "extractOnly=true&wt=json&indent=true" --out yes example/exampledocs/sample.html
+$ bin/solr post -c gettingstarted --params "extractOnly=true&wt=json&indent=true" --verbose example/exampledocs/sample.html
 ----
 
 === Using Solr Cell with a POST Request
diff --git a/solr/solr-ref-guide/modules/indexing-guide/pages/post-tool.adoc b/solr/solr-ref-guide/modules/indexing-guide/pages/post-tool.adoc
index 97f28887d16..4483557dd29 100644
--- a/solr/solr-ref-guide/modules/indexing-guide/pages/post-tool.adoc
+++ b/solr/solr-ref-guide/modules/indexing-guide/pages/post-tool.adoc
@@ -41,8 +41,8 @@ The basic usage of `bin/solr post` is:
 
 [source,plain]
 ----
-usage: bin/solr post [-c <NAME>] [-d <delay>] [--dry-run] [--format] [-ft <<type>[,<type>,...]>] [-h] [--mode <mode>]
-       [-o] [--out] [--params <<key>=<value>[&<key>=<value>...]>] [-r <recursive>] [-s <HOST>] [--skip-commit] [-t
+usage: bin/solr post [-c <NAME>] [--delay <delay>] [--dry-run] [--format] [-ft <<type>[,<type>,...]>] [-h] [--mode
+       <mode>] [-o] [--params <<key>=<value>[&<key>=<value>...]>] [-r <recursive>] [-s <HOST>] [--skip-commit] [-t
        <content-type>] [-u <credentials>] [--verbose]
 
 List of options:
@@ -61,7 +61,6 @@ List of options:
                                                   directory, 'web' crawls website, 'args' processes input args, and
                                                   'stdin' reads a command from standard in. default: files.
  -o,--optimize                                    Issue an optimize at end of posting documents.
-    --out                                         sends Solr response outputs to console.
     --params <<key>=<value>[&<key>=<value>...]>   Values must be URL-encoded; these pass through to Solr update request.
  -r,--recursive <recursive>                       For web crawl, how deep to go. default: 1
  -s,--solr-url <HOST>                             Base Solr URL, which can be used to determine the zk-host if that's
@@ -177,11 +176,11 @@ $ bin/solr post --mode web -c gettingstarted --recursive 1 --delay 1 https://sol
 === Standard Input as Source for Indexing
 
 You can use the standard input as your source for data to index.  
-Notice the `-out` providing raw responses from Solr.
+Notice the `--verbose` providing raw responses from Solr.
 
 [,console]
 ----
-$ echo '{commit: {}}' | bin/solr post --mode stdin --solr-url http://localhost:8983 --name my_collection --out
+$ echo '{commit: {}}' | bin/solr post --mode stdin --solr-url http://localhost:8983 --name my_collection --verbose
 ----
 
 === Raw Data as Source for Indexing
@@ -190,5 +189,5 @@ Provide the raw document as a string for indexing.
 
 [,console]
 ----
-$ bin/solr post --solr-url http://localhost:8983 --name signals -mode args --type text/csv -out $'id,value\n1,0.47' 
+$ bin/solr post --solr-url http://localhost:8983 --name signals -mode args --type text/csv --verbose $'id,value\n1,0.47' 
 ----

From 3842aca96ad1021f59b1e0f8f704c065f2afdf2f Mon Sep 17 00:00:00 2001
From: Eric Pugh <epugh@opensourceconnections.com>
Date: Tue, 29 Oct 2024 12:52:14 -0400
Subject: [PATCH 05/32] Align with linux (#2817)

---
 solr/bin/solr.cmd | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/solr/bin/solr.cmd b/solr/bin/solr.cmd
index d6ff9e62492..d6b6f18e248 100755
--- a/solr/bin/solr.cmd
+++ b/solr/bin/solr.cmd
@@ -1143,9 +1143,8 @@ IF "%FG%"=="1" (
     -Djava.io.tmpdir="%SOLR_SERVER_DIR%\tmp" -jar start.jar %SOLR_JETTY_CONFIG% "%SOLR_JETTY_ADDL_CONFIG%" > "!SOLR_LOGS_DIR!\solr-%SOLR_PORT%-console.log"
   echo %SOLR_PORT%>"%SOLR_TIP%"\bin\solr-%SOLR_PORT%.port
 
-  REM default to 30 seconds for backwards compatibility.
   IF "!SOLR_START_WAIT!"=="" (
-    set SOLR_START_WAIT=30
+    set SOLR_START_WAIT=180
   )
   REM now wait to see Solr come online ...
   "%JAVA%" %SOLR_SSL_OPTS% %AUTHC_OPTS% %SOLR_ZK_CREDS_AND_ACLS% %SOLR_TOOL_OPTS% -Dsolr.install.dir="%SOLR_TIP%" -Dsolr.default.confdir="%DEFAULT_CONFDIR%"^

From 700d2f2291030c0eade9b20004fdd428e633b7c1 Mon Sep 17 00:00:00 2001
From: Jason Gerlowski <gerlowskija@apache.org>
Date: Tue, 29 Oct 2024 11:15:22 -0700
Subject: [PATCH 06/32] SOLR-17515: Always init Http2SolrClient authStore
 (#2802)

Prior to this commit, Http2SolrClient only initialized its
'authenticationStore' instance var when creating a fresh Jetty
HttpClient.  This causes NPE's later on when client code (e.g.
PreemptiveBasicAuthClientBuilderFactory) would attempt to access the
object.

This commit fixes this by making sure that 'authenticationStore' is
initialized when using an existing Jetty client (typically by reusing
the AuthenticationStore already used by the client).
---
 .../org/apache/solr/cloud/RecoveryZkTestWithAuth.java  | 10 ++++++++++
 .../apache/solr/client/solrj/impl/Http2SolrClient.java | 10 ++++++++++
 2 files changed, 20 insertions(+)

diff --git a/solr/core/src/test/org/apache/solr/cloud/RecoveryZkTestWithAuth.java b/solr/core/src/test/org/apache/solr/cloud/RecoveryZkTestWithAuth.java
index 63f66736b83..2cec10d8610 100644
--- a/solr/core/src/test/org/apache/solr/cloud/RecoveryZkTestWithAuth.java
+++ b/solr/core/src/test/org/apache/solr/cloud/RecoveryZkTestWithAuth.java
@@ -27,6 +27,7 @@
 import org.apache.solr.client.solrj.SolrResponse;
 import org.apache.solr.client.solrj.SolrServerException;
 import org.apache.solr.client.solrj.impl.CloudLegacySolrClient;
+import org.apache.solr.client.solrj.impl.HttpClientUtil;
 import org.apache.solr.client.solrj.impl.HttpSolrClient;
 import org.apache.solr.client.solrj.request.CollectionAdminRequest;
 import org.apache.solr.client.solrj.request.QueryRequest;
@@ -43,6 +44,15 @@
 public class RecoveryZkTestWithAuth extends SolrCloudTestCase {
   @BeforeClass
   public static void setupCluster() throws Exception {
+    // Periodically mimic the sysprops set by bin/solr when it knows auth is active (see SOLR-17515
+    // for context)
+    if (rarely()) {
+      System.setProperty(
+          HttpClientUtil.SYS_PROP_HTTP_CLIENT_BUILDER_FACTORY,
+          "org.apache.solr.client.solrj.impl.PreemptiveBasicAuthClientBuilderFactory");
+      System.setProperty("basicauth", SecurityJson.USER_PASS);
+    }
+
     cluster =
         configureCluster(1)
             .addConfig("conf", configset("cloud-minimal"))
diff --git a/solr/solrj/src/java/org/apache/solr/client/solrj/impl/Http2SolrClient.java b/solr/solrj/src/java/org/apache/solr/client/solrj/impl/Http2SolrClient.java
index c35e11cf491..e3f974e41e3 100644
--- a/solr/solrj/src/java/org/apache/solr/client/solrj/impl/Http2SolrClient.java
+++ b/solr/solrj/src/java/org/apache/solr/client/solrj/impl/Http2SolrClient.java
@@ -133,6 +133,8 @@ protected Http2SolrClient(String serverBaseUrl, Builder builder) {
       if (this.executor == null) {
         this.executor = builder.executor;
       }
+
+      initAuthStoreFromExistingClient(httpClient);
       this.closeClient = false;
     } else {
       this.httpClient = createHttpClient(builder);
@@ -148,6 +150,14 @@ protected Http2SolrClient(String serverBaseUrl, Builder builder) {
     assert ObjectReleaseTracker.track(this);
   }
 
+  private void initAuthStoreFromExistingClient(HttpClient httpClient) {
+    // Since we don't allow users to provide arbitrary Jetty clients, all parameters to this method
+    // must originate from the 'createHttpClient' method, which uses AuthenticationStoreHolder.
+    // Verify this assumption and copy the existing instance to avoid unnecessary wrapping.
+    assert httpClient.getAuthenticationStore() instanceof AuthenticationStoreHolder;
+    this.authenticationStore = (AuthenticationStoreHolder) httpClient.getAuthenticationStore();
+  }
+
   @Deprecated(since = "9.7")
   public void addListenerFactory(HttpListenerFactory factory) {
     this.listenerFactory.add(factory);

From 8f18729f97e897f3755421c65b4b539fbc871746 Mon Sep 17 00:00:00 2001
From: David Smiley <dsmiley@apache.org>
Date: Tue, 29 Oct 2024 21:26:05 -0400
Subject: [PATCH 07/32] SOLR-17485 Remove CheckLoggingConfiguration;
 unnecessary (#2763)

A refactoring, and removing needless CheckLoggingConfiguration.

Misc changes:
* Extend HttpFilter etc.; remove needless subclasses.  Thus don't need to cast HttpServletRequest.
* remove no-op needless destroy(); same as super
* Move excludePath check to very first thing
* rename doFilter(...retry) to doFilterRetry
* move closeShield to before doFilterRetry so it doesn't have to know about retry, simplifying things.
* no longer have leading underscore param names since no longer have variable masking
---
 .../apache/solr/servlet/BaseSolrFilter.java   | 31 ------
 .../apache/solr/servlet/BaseSolrServlet.java  | 32 -------
 .../servlet/CheckLoggingConfiguration.java    | 37 -------
 .../solr/servlet/LoadAdminUiServlet.java      | 11 ++-
 .../apache/solr/servlet/RedirectServlet.java  |  3 +-
 .../org/apache/solr/servlet/ServletUtils.java | 96 ++++++++-----------
 .../solr/servlet/SolrDispatchFilter.java      | 32 +++----
 7 files changed, 59 insertions(+), 183 deletions(-)
 delete mode 100644 solr/core/src/java/org/apache/solr/servlet/BaseSolrFilter.java
 delete mode 100644 solr/core/src/java/org/apache/solr/servlet/BaseSolrServlet.java
 delete mode 100644 solr/core/src/java/org/apache/solr/servlet/CheckLoggingConfiguration.java

diff --git a/solr/core/src/java/org/apache/solr/servlet/BaseSolrFilter.java b/solr/core/src/java/org/apache/solr/servlet/BaseSolrFilter.java
deleted file mode 100644
index d16d722447d..00000000000
--- a/solr/core/src/java/org/apache/solr/servlet/BaseSolrFilter.java
+++ /dev/null
@@ -1,31 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements.  See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License.  You may obtain a copy of the License at
- *
- *     http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.solr.servlet;
-
-import javax.servlet.Filter;
-
-/**
- * All Solr filters available to the user's webapp should extend this class and not just implement
- * {@link Filter}. This class ensures that the logging configuration is correct before any Solr
- * specific code is executed.
- */
-abstract class BaseSolrFilter implements Filter {
-
-  static {
-    CheckLoggingConfiguration.check();
-  }
-}
diff --git a/solr/core/src/java/org/apache/solr/servlet/BaseSolrServlet.java b/solr/core/src/java/org/apache/solr/servlet/BaseSolrServlet.java
deleted file mode 100644
index 85aca08cd86..00000000000
--- a/solr/core/src/java/org/apache/solr/servlet/BaseSolrServlet.java
+++ /dev/null
@@ -1,32 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements.  See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License.  You may obtain a copy of the License at
- *
- *     http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.solr.servlet;
-
-import javax.servlet.http.HttpServlet;
-
-/**
- * All Solr servlets available to the user's webapp should extend this class and not {@link
- * HttpServlet}. This class ensures that the logging configuration is correct before any Solr
- * specific code is executed.
- */
-@SuppressWarnings("serial")
-abstract class BaseSolrServlet extends HttpServlet {
-
-  static {
-    CheckLoggingConfiguration.check();
-  }
-}
diff --git a/solr/core/src/java/org/apache/solr/servlet/CheckLoggingConfiguration.java b/solr/core/src/java/org/apache/solr/servlet/CheckLoggingConfiguration.java
deleted file mode 100644
index 70506dea12e..00000000000
--- a/solr/core/src/java/org/apache/solr/servlet/CheckLoggingConfiguration.java
+++ /dev/null
@@ -1,37 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements.  See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License.  You may obtain a copy of the License at
- *
- *     http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.solr.servlet;
-
-import java.lang.invoke.MethodHandles;
-import org.slf4j.LoggerFactory;
-
-final class CheckLoggingConfiguration {
-
-  static void check() {
-    try {
-      LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
-    } catch (NoClassDefFoundError e) {
-      throw new NoClassDefFoundError(
-          "Failed to initialize Apache Solr: "
-              + "Could not find necessary SLF4j logging jars. If using Jetty, the SLF4j logging jars need to go in "
-              + "the jetty lib/ext directory. For other containers, the corresponding directory should be used. "
-              + "For more information, see: https://cwiki.apache.org/confluence/display/solr/SolrLogging#SolrLogging-UsingtheexampleloggingsetupincontainersotherthanJetty");
-    }
-  }
-
-  private CheckLoggingConfiguration() {}
-}
diff --git a/solr/core/src/java/org/apache/solr/servlet/LoadAdminUiServlet.java b/solr/core/src/java/org/apache/solr/servlet/LoadAdminUiServlet.java
index 9da4bc84775..436906d3a69 100644
--- a/solr/core/src/java/org/apache/solr/servlet/LoadAdminUiServlet.java
+++ b/solr/core/src/java/org/apache/solr/servlet/LoadAdminUiServlet.java
@@ -25,6 +25,7 @@
 import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.List;
+import javax.servlet.http.HttpServlet;
 import javax.servlet.http.HttpServletRequest;
 import javax.servlet.http.HttpServletResponse;
 import org.apache.commons.io.output.CloseShieldOutputStream;
@@ -36,7 +37,7 @@
  *
  * @since solr 4.0
  */
-public final class LoadAdminUiServlet extends BaseSolrServlet {
+public final class LoadAdminUiServlet extends HttpServlet {
 
   // check system properties for whether or not admin UI is disabled, default is false
   private static final boolean disabled =
@@ -45,16 +46,16 @@ public final class LoadAdminUiServlet extends BaseSolrServlet {
   public static final String SYSPROP_CSP_CONNECT_SRC_URLS = "solr.ui.headers.csp.connect-src.urls";
 
   @Override
-  public void doGet(HttpServletRequest _request, HttpServletResponse _response) throws IOException {
+  public void doGet(HttpServletRequest request, HttpServletResponse response) throws IOException {
     if (disabled) {
-      _response.sendError(
+      response.sendError(
           404,
           "Solr Admin UI is disabled. To enable it, change the default value of SOLR_ADMIN_UI_"
               + "ENABLED in bin/solr.in.sh or solr.in.cmd.");
       return;
     }
-    HttpServletRequest request = ServletUtils.closeShield(_request, false);
-    HttpServletResponse response = ServletUtils.closeShield(_response, false);
+    request = ServletUtils.closeShield(request);
+    response = ServletUtils.closeShield(response);
 
     response.addHeader(
         "X-Frame-Options", "DENY"); // security: SOLR-7966 - avoid clickjacking for admin interface
diff --git a/solr/core/src/java/org/apache/solr/servlet/RedirectServlet.java b/solr/core/src/java/org/apache/solr/servlet/RedirectServlet.java
index a67e20831de..a0c11b15a04 100644
--- a/solr/core/src/java/org/apache/solr/servlet/RedirectServlet.java
+++ b/solr/core/src/java/org/apache/solr/servlet/RedirectServlet.java
@@ -19,11 +19,12 @@
 import java.io.IOException;
 import javax.servlet.ServletConfig;
 import javax.servlet.ServletException;
+import javax.servlet.http.HttpServlet;
 import javax.servlet.http.HttpServletRequest;
 import javax.servlet.http.HttpServletResponse;
 
 /** A Simple redirection servlet to help us deprecate old UI elements */
-public class RedirectServlet extends BaseSolrServlet {
+public class RedirectServlet extends HttpServlet {
 
   static final String CONTEXT_KEY = "${context}";
 
diff --git a/solr/core/src/java/org/apache/solr/servlet/ServletUtils.java b/solr/core/src/java/org/apache/solr/servlet/ServletUtils.java
index 605f1c6c668..f9be5246b3d 100644
--- a/solr/core/src/java/org/apache/solr/servlet/ServletUtils.java
+++ b/solr/core/src/java/org/apache/solr/servlet/ServletUtils.java
@@ -72,78 +72,62 @@ public static String getPathAfterContext(HttpServletRequest request) {
   }
 
   /**
-   * Wrap the request's input stream with a close shield. If this is a retry, we will assume that
-   * the stream has already been wrapped and do nothing.
+   * Wrap the request's input stream with a close shield.
    *
    * <p>Only the container should ever actually close the servlet output stream. This method
    * possibly should be turned into a servlet filter
    *
    * @param request The request to wrap.
-   * @param retry If this is an original request or a retry.
    * @return A request object with an {@link InputStream} that will ignore calls to close.
    */
-  public static HttpServletRequest closeShield(HttpServletRequest request, boolean retry) {
-    if (!retry) {
-      return new HttpServletRequestWrapper(request) {
-
-        @Override
-        public ServletInputStream getInputStream() throws IOException {
-
-          return new ServletInputStreamWrapper(super.getInputStream()) {
-            @Override
-            public void close() {
-              // even though we skip closes, we let local tests know not to close so that a full
-              // understanding can take place
-              assert !Thread.currentThread()
-                      .getStackTrace()[2]
-                      .getClassName()
-                      .matches("org\\.apache\\.(?:solr|lucene).*")
-                  : CLOSE_STREAM_MSG;
-              this.stream = ClosedServletInputStream.CLOSED_SERVLET_INPUT_STREAM;
-            }
-          };
-        }
-      };
-    } else {
-      return request;
-    }
+  public static HttpServletRequest closeShield(HttpServletRequest request) {
+    return new HttpServletRequestWrapper(request) {
+      @Override
+      public ServletInputStream getInputStream() throws IOException {
+        return new ServletInputStreamWrapper(super.getInputStream()) {
+          @Override
+          public void close() {
+            // even though we skip closes, we let local tests know not to close so that a full
+            // understanding can take place
+            assert !Thread.currentThread()
+                    .getStackTrace()[2]
+                    .getClassName()
+                    .matches("org\\.apache\\.(?:solr|lucene).*")
+                : CLOSE_STREAM_MSG;
+            this.stream = ClosedServletInputStream.CLOSED_SERVLET_INPUT_STREAM;
+          }
+        };
+      }
+    };
   }
 
   /**
-   * Wrap the response's output stream with a close shield. If this is a retry, we will assume that
-   * the stream has already been wrapped and do nothing.
+   * Wrap the response's output stream with a close shield.
    *
    * <p>Only the container should ever actually close the servlet request stream.
    *
    * @param response The response to wrap.
-   * @param retry If this response corresponds to an original request or a retry.
    * @return A response object with an {@link OutputStream} that will ignore calls to close.
    */
-  public static HttpServletResponse closeShield(HttpServletResponse response, boolean retry) {
-    if (!retry) {
-      return new HttpServletResponseWrapper(response) {
-
-        @Override
-        public ServletOutputStream getOutputStream() throws IOException {
-
-          return new ServletOutputStreamWrapper(super.getOutputStream()) {
-            @Override
-            public void close() {
-              // even though we skip closes, we let local tests know not to close so that a full
-              // understanding can take place
-              assert !Thread.currentThread()
-                      .getStackTrace()[2]
-                      .getClassName()
-                      .matches("org\\.apache\\.(?:solr|lucene).*")
-                  : CLOSE_STREAM_MSG;
-              stream = ClosedServletOutputStream.CLOSED_SERVLET_OUTPUT_STREAM;
-            }
-          };
-        }
-      };
-    } else {
-      return response;
-    }
+  public static HttpServletResponse closeShield(HttpServletResponse response) {
+    return new HttpServletResponseWrapper(response) {
+      @Override
+      public ServletOutputStream getOutputStream() throws IOException {
+        return new ServletOutputStreamWrapper(super.getOutputStream()) {
+          @Override
+          public void close() {
+            // even though we skip closes, we let local tests know not to close so that a full
+            // understanding can take place
+            assert !Thread.currentThread()
+                    .getStackTrace()[2]
+                    .getClassName()
+                    .matches("org\\.apache\\.(?:solr|lucene).*")
+                : CLOSE_STREAM_MSG;
+            stream = ClosedServletOutputStream.CLOSED_SERVLET_OUTPUT_STREAM;
+          }
+        };
+      }
+    };
   }
 
   static boolean excludedPath(
diff --git a/solr/core/src/java/org/apache/solr/servlet/SolrDispatchFilter.java b/solr/core/src/java/org/apache/solr/servlet/SolrDispatchFilter.java
index 5cda46ddcb3..189ebeec134 100644
--- a/solr/core/src/java/org/apache/solr/servlet/SolrDispatchFilter.java
+++ b/solr/core/src/java/org/apache/solr/servlet/SolrDispatchFilter.java
@@ -33,9 +33,8 @@
 import javax.servlet.FilterChain;
 import javax.servlet.FilterConfig;
 import javax.servlet.ServletException;
-import javax.servlet.ServletRequest;
-import javax.servlet.ServletResponse;
 import javax.servlet.UnavailableException;
+import javax.servlet.http.HttpFilter;
 import javax.servlet.http.HttpServletRequest;
 import javax.servlet.http.HttpServletResponse;
 import org.apache.http.client.HttpClient;
@@ -69,7 +68,7 @@
 // servlets that are more focused in scope. This should become possible now that we have a
 // ServletContextListener for startup/shutdown of CoreContainer that sets up a service from which
 // things like CoreContainer can be requested. (or better yet injected)
-public class SolrDispatchFilter extends BaseSolrFilter implements PathExcluder {
+public class SolrDispatchFilter extends HttpFilter implements PathExcluder {
   private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
 
   private CoreContainerProvider containerProvider;
@@ -160,38 +159,29 @@ public CoreContainer getCores() throws UnavailableException {
     return containerProvider.getCoreContainer();
   }
 
-  @Override
-  public void destroy() {
-    // CoreService shuts itself down as a ContextListener. The filter does not own anything with a
-    // lifecycle anymore! Yay!
-  }
-
   @Override
   @SuppressForbidden(
       reason =
           "Set the thread contextClassLoader for all 3rd party dependencies that we cannot control")
-  public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain)
+  public void doFilter(HttpServletRequest request, HttpServletResponse response, FilterChain chain)
       throws IOException, ServletException {
+    if (excludedPath(excludePatterns, request, response, chain)) {
+      return;
+    }
+
     try (var mdcSnapshot = MDCSnapshot.create()) {
       assert null != mdcSnapshot; // prevent compiler warning
       MDCLoggingContext.reset();
       MDCLoggingContext.setNode(getCores());
       Thread.currentThread().setContextClassLoader(getCores().getResourceLoader().getClassLoader());
 
-      doFilter(request, response, chain, false);
+      doFilterRetry(closeShield(request), closeShield(response), chain, false);
     }
   }
 
-  public void doFilter(
-      ServletRequest _request, ServletResponse _response, FilterChain chain, boolean retry)
+  protected void doFilterRetry(
+      HttpServletRequest request, HttpServletResponse response, FilterChain chain, boolean retry)
       throws IOException, ServletException {
-    if (!(_request instanceof HttpServletRequest)) return;
-    HttpServletRequest request = closeShield((HttpServletRequest) _request, retry);
-    HttpServletResponse response = closeShield((HttpServletResponse) _response, retry);
-
-    if (excludedPath(excludePatterns, request, response, chain)) {
-      return;
-    }
     setTracer(request, getCores().getTracer());
     RateLimitManager rateLimitManager = containerProvider.getRateLimitManager();
     try {
@@ -248,7 +238,7 @@ private void dispatch(
           break;
         case RETRY:
           span.addEvent("SolrDispatchFilter RETRY");
-          doFilter(request, response, chain, true); // RECURSION
+          doFilterRetry(request, response, chain, true); // RECURSION
           break;
         case FORWARD:
           span.addEvent("SolrDispatchFilter FORWARD");

From e377ebbdaff5ae784b3f4e4315afa4d6705d1d71 Mon Sep 17 00:00:00 2001
From: Zack Kendall <zachariahkendall@gmail.com>
Date: Tue, 29 Oct 2024 18:29:41 -0700
Subject: [PATCH 08/32] SOLR-11191: SolrIndexSplitter: Routing child docs by
 _root_ when available (#2799)

Splitting shards now supports child / nested docs.  Works based on splitting based on the _root_ field.


Co-authored-by: zkendall <zkendall@axon.com>
---
 solr/CHANGES.txt                              |   2 +
 .../apache/solr/update/SolrIndexSplitter.java |  15 +-
 .../configsets/cloud-minimal/conf/schema.xml  |   2 +
 .../org/apache/solr/cloud/SplitShardTest.java |  71 +++++++---
 .../solr/update/SolrIndexSplitterTest.java    | 131 +++++++++++++++++-
 .../pages/collection-management.adoc          |   1 +
 .../deployment-guide/pages/node-roles.adoc    |   2 +-
 .../pages/solrcloud-shards-indexing.adoc      |   2 +-
 .../pages/indexing-nested-documents.adoc      |   6 +-
 9 files changed, 208 insertions(+), 24 deletions(-)

diff --git a/solr/CHANGES.txt b/solr/CHANGES.txt
index ac00cf68152..db44d16aa74 100644
--- a/solr/CHANGES.txt
+++ b/solr/CHANGES.txt
@@ -189,6 +189,8 @@ Bug Fixes
 
 * SOLR-17413: Fixed UpdateLog replay bug that shared thread-unsafe SolrQueryRequest objects across threads (Jason Gerlowski, David Smiley, Houston Putman)
 
+* SOLR-11191: Splitting shards now routes child-docs with their _root_ field when available so they maintain parent relationship. (Zack Kendall)
+
 Dependency Upgrades
 ---------------------
 (No changes)
diff --git a/solr/core/src/java/org/apache/solr/update/SolrIndexSplitter.java b/solr/core/src/java/org/apache/solr/update/SolrIndexSplitter.java
index 61ba505ddd5..fa12d7b51d4 100644
--- a/solr/core/src/java/org/apache/solr/update/SolrIndexSplitter.java
+++ b/solr/core/src/java/org/apache/solr/update/SolrIndexSplitter.java
@@ -69,6 +69,7 @@
 import org.apache.solr.core.SolrCore;
 import org.apache.solr.handler.IndexFetcher;
 import org.apache.solr.handler.SnapShooter;
+import org.apache.solr.schema.IndexSchema;
 import org.apache.solr.schema.SchemaField;
 import org.apache.solr.search.BitsFilteredPostingsEnum;
 import org.apache.solr.search.SolrIndexSearcher;
@@ -125,17 +126,29 @@ public SolrIndexSplitter(SplitIndexCommand cmd) {
       numPieces = cmd.ranges.size();
       rangesArr = cmd.ranges.toArray(new DocRouter.Range[0]);
     }
+
     if (cmd.routeFieldName == null) {
-      field = searcher.getSchema().getUniqueKeyField();
+      // To support routing child documents, use the root field if it exists (which would be
+      // populated with unique field), otherwise use the unique key field
+      if (searcher.getSchema().isUsableForChildDocs()) {
+        field = searcher.getSchema().getField(IndexSchema.ROOT_FIELD_NAME);
+      } else {
+        field = searcher.getSchema().getUniqueKeyField();
+      }
     } else {
+      // Custom routing
+      // If child docs are used, users must ensure that the whole nested document tree has a
+      // consistent routeField value
       field = searcher.getSchema().getField(cmd.routeFieldName);
     }
+
     if (cmd.splitKey == null) {
       splitKey = null;
     } else {
       checkRouterSupportsSplitKey(hashRouter, cmd.splitKey);
       splitKey = ((CompositeIdRouter) hashRouter).getRouteKeyNoSuffix(cmd.splitKey);
     }
+
     if (cmd.cores == null) {
       this.splitMethod = SplitMethod.REWRITE;
     } else {
diff --git a/solr/core/src/test-files/solr/configsets/cloud-minimal/conf/schema.xml b/solr/core/src/test-files/solr/configsets/cloud-minimal/conf/schema.xml
index fc23706512e..3c25f27405a 100644
--- a/solr/core/src/test-files/solr/configsets/cloud-minimal/conf/schema.xml
+++ b/solr/core/src/test-files/solr/configsets/cloud-minimal/conf/schema.xml
@@ -19,10 +19,12 @@
   <fieldType name="string" class="solr.StrField"/>
   <fieldType name="int" class="${solr.tests.IntegerFieldType}" docValues="${solr.tests.numeric.dv}" precisionStep="0" omitNorms="true" positionIncrementGap="0" uninvertible="true"/>
   <fieldType name="long" class="${solr.tests.LongFieldType}" docValues="${solr.tests.numeric.dv}" precisionStep="0" omitNorms="true" positionIncrementGap="0" uninvertible="true"/>
+  <fieldType name="_nest_path_" class="solr.NestPathField"/>
   <dynamicField name="*" type="string" indexed="true" stored="true"/>
   <!-- for versioning -->
   <field name="_version_" type="long" indexed="true" stored="true"/>
   <field name="_root_" type="string" indexed="true" stored="true" multiValued="false" required="false"/>
+  <field name="_nest_path_" type="_nest_path_"/>`
   <field name="id" type="string" indexed="true" stored="true"/>
   <dynamicField name="*_s"  type="string"  indexed="true"  stored="true" />
   <uniqueKey>id</uniqueKey>
diff --git a/solr/core/src/test/org/apache/solr/cloud/SplitShardTest.java b/solr/core/src/test/org/apache/solr/cloud/SplitShardTest.java
index 132dfcfaaa2..ab194b291da 100644
--- a/solr/core/src/test/org/apache/solr/cloud/SplitShardTest.java
+++ b/solr/core/src/test/org/apache/solr/cloud/SplitShardTest.java
@@ -21,6 +21,7 @@
 
 import java.io.IOException;
 import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
 import java.util.Collection;
 import java.util.HashMap;
 import java.util.HashSet;
@@ -35,11 +36,14 @@
 import org.apache.solr.client.solrj.SolrServerException;
 import org.apache.solr.client.solrj.impl.CloudSolrClient;
 import org.apache.solr.client.solrj.request.CollectionAdminRequest;
+import org.apache.solr.client.solrj.request.QueryRequest;
 import org.apache.solr.client.solrj.request.UpdateRequest;
+import org.apache.solr.client.solrj.response.QueryResponse;
 import org.apache.solr.client.solrj.response.UpdateResponse;
 import org.apache.solr.common.SolrDocument;
 import org.apache.solr.common.SolrDocumentList;
 import org.apache.solr.common.SolrException;
+import org.apache.solr.common.SolrInputDocument;
 import org.apache.solr.common.cloud.DocCollection;
 import org.apache.solr.common.cloud.Replica;
 import org.apache.solr.common.cloud.Slice;
@@ -88,16 +92,7 @@ public void doTest() throws IOException, SolrServerException {
             .setNumSubShards(5)
             .setShardName("shard1");
     splitShard.process(cluster.getSolrClient());
-    waitForState(
-        "Timed out waiting for sub shards to be active. Number of active shards="
-            + cluster
-                .getSolrClient()
-                .getClusterState()
-                .getCollection(COLLECTION_NAME)
-                .getActiveSlices()
-                .size(),
-        COLLECTION_NAME,
-        activeClusterShape(6, 6));
+    waitForState("Waiting for 6 shards after split", COLLECTION_NAME, activeClusterShape(6, 6));
 
     try {
       splitShard =
@@ -153,15 +148,7 @@ public void testSplitFuzz() throws Exception {
         CollectionAdminRequest.splitShard(collectionName).setSplitFuzz(0.5f).setShardName("shard1");
     splitShard.process(cluster.getSolrClient());
     waitForState(
-        "Timed out waiting for sub shards to be active. Number of active shards="
-            + cluster
-                .getSolrClient()
-                .getClusterState()
-                .getCollection(collectionName)
-                .getActiveSlices()
-                .size(),
-        collectionName,
-        activeClusterShape(3, 3));
+        "Waiting for 3 active shards after split", collectionName, activeClusterShape(3, 3));
     DocCollection coll = cluster.getSolrClient().getClusterState().getCollection(collectionName);
     Slice s1_0 = coll.getSlice("shard1_0");
     Slice s1_1 = coll.getSlice("shard1_1");
@@ -174,6 +161,52 @@ public void testSplitFuzz() throws Exception {
     assertEquals("wrong range in s1_1", expected1, delta1);
   }
 
+  @Test
+  public void testWithChildDocuments() throws Exception {
+    CloudSolrClient solrClient = cluster.getSolrClient();
+    CollectionAdminRequest.createCollection(COLLECTION_NAME, "conf", 1, 1).process(solrClient);
+
+    cluster.waitForActiveCollection(COLLECTION_NAME, 1, 1);
+
+    List<SolrInputDocument> inputDocs = new ArrayList<>();
+    for (int idx = 0; idx < 20; ) { // 10 parents + 10 children
+      SolrInputDocument parent = new SolrInputDocument();
+      parent.addField("id", ++idx);
+      parent.addField("type_s", "parent");
+
+      // Child has different ID, so could be routed differently if used for routing.
+      SolrInputDocument child = new SolrInputDocument();
+      child.addField("id", ++idx);
+      child.addField("expected_parent_s", parent.getField("id"));
+
+      parent.addField("myChild", List.of(child));
+
+      inputDocs.add(parent);
+    }
+    solrClient.add(COLLECTION_NAME, inputDocs);
+
+    CollectionAdminRequest.SplitShard splitShard =
+        CollectionAdminRequest.splitShard(COLLECTION_NAME)
+            .setNumSubShards(2)
+            .setShardName("shard1");
+    splitShard.setWaitForFinalState(true);
+    splitShard.process(solrClient);
+    waitForState(
+        "Waiting for 2 active shards after split", COLLECTION_NAME, activeClusterShape(2, 2));
+
+    QueryRequest req =
+        new QueryRequest(new SolrQuery("type_s:parent").setFields("*", "[child]", "[shard]"));
+    QueryResponse rsp = req.process(solrClient, COLLECTION_NAME);
+    assertEquals(10, rsp.getResults().getNumFound());
+    for (SolrDocument doc : rsp.getResults()) {
+      @SuppressWarnings("unchecked")
+      List<SolrDocument> children = (List<SolrDocument>) doc.get("myChild");
+      assertEquals("only 1 child per parent", 1, children.size());
+      SolrDocument child = children.get(0);
+      assertEquals("parent-child match", doc.get("id"), child.get("expected_parent_s"));
+    }
+  }
+
   private CloudSolrClient createCollection(String collectionName, int repFactor) throws Exception {
 
     CollectionAdminRequest.createCollection(collectionName, "conf", 1, repFactor)
diff --git a/solr/core/src/test/org/apache/solr/update/SolrIndexSplitterTest.java b/solr/core/src/test/org/apache/solr/update/SolrIndexSplitterTest.java
index 2796233ea0b..78906e9b8e0 100644
--- a/solr/core/src/test/org/apache/solr/update/SolrIndexSplitterTest.java
+++ b/solr/core/src/test/org/apache/solr/update/SolrIndexSplitterTest.java
@@ -28,6 +28,7 @@
 import org.apache.solr.SolrTestCaseJ4;
 import org.apache.solr.client.solrj.SolrQuery;
 import org.apache.solr.client.solrj.embedded.EmbeddedSolrServer;
+import org.apache.solr.common.SolrInputDocument;
 import org.apache.solr.common.cloud.CompositeIdRouter;
 import org.apache.solr.common.cloud.DocRouter;
 import org.apache.solr.common.cloud.PlainIdRouter;
@@ -485,19 +486,147 @@ private void doTestSplitByRouteKey(SolrIndexSplitter.SplitMethod splitMethod) th
     }
   }
 
+  @Test
+  public void testSplitWithChildDocs() throws Exception {
+    doTestSplitWithChildDocs(SolrIndexSplitter.SplitMethod.REWRITE);
+  }
+
+  @Test
+  public void testSplitWithChildDocsLink() throws Exception {
+    doTestSplitWithChildDocs(SolrIndexSplitter.SplitMethod.LINK);
+  }
+
+  public void doTestSplitWithChildDocs(SolrIndexSplitter.SplitMethod splitMethod) throws Exception {
+    // Overall test/split pattern copied from doTestSplitByCores
+    File indexDir = createTempDir().toFile();
+
+    CompositeIdRouter r1 = new CompositeIdRouter();
+    String routeKeyBase = "sea-line!";
+
+    List<DocRouter.Range> twoRanges = r1.partitionRange(2, r1.keyHashRange(routeKeyBase));
+
+    // Insert other doc for contrast
+    String otherDocId = routeKeyBase + "6"; // Will hash into first range
+    assertU(adoc("id", otherDocId));
+
+    // Insert child doc
+    String parentId = routeKeyBase + "1"; // Will hash into second range
+    String childId = routeKeyBase + "5"; // Will hash into first range
+    SolrInputDocument doc =
+        sdoc("id", parentId, "myChild", sdocs(sdoc("id", childId, "child_s", "child")));
+    assertU(adoc(doc));
+
+    assertU(commit());
+    assertJQ(req("q", "*:*"), "/response/numFound==3");
+
+    // Able to query child.
+    assertJQ(
+        req("q", "id:" + parentId, "fl", "*, [child]"),
+        "/response/numFound==1",
+        "/response/docs/[0]/myChild/[0]/id=='" + childId + "'",
+        "/response/docs/[0]/myChild/[0]/_root_=='"
+            + parentId
+            + "'" // Child has parent root to route with
+        );
+
+    SolrCore core1 = null, core2 = null;
+    try {
+      core1 =
+          h.getCoreContainer()
+              .create(
+                  "split1",
+                  Map.of("dataDir", indexDir1.getAbsolutePath(), "configSet", "cloud-minimal"));
+      core2 =
+          h.getCoreContainer()
+              .create(
+                  "split2",
+                  Map.of("dataDir", indexDir2.getAbsolutePath(), "configSet", "cloud-minimal"));
+
+      LocalSolrQueryRequest request = null;
+      try {
+        request = lrf.makeRequest("q", "dummy");
+        SolrQueryResponse rsp = new SolrQueryResponse();
+        SplitIndexCommand command =
+            new SplitIndexCommand(
+                request,
+                rsp,
+                null,
+                List.of(core1, core2),
+                twoRanges,
+                new CompositeIdRouter(),
+                null,
+                null,
+                splitMethod);
+        doSplit(command);
+      } finally {
+        if (request != null) request.close();
+      }
+      @SuppressWarnings("resource")
+      final EmbeddedSolrServer server1 = new EmbeddedSolrServer(h.getCoreContainer(), "split1");
+      @SuppressWarnings("resource")
+      final EmbeddedSolrServer server2 = new EmbeddedSolrServer(h.getCoreContainer(), "split2");
+      server1.commit(true, true);
+      server2.commit(true, true);
+      assertEquals(
+          "should be  2 docs in index2",
+          2,
+          server2.query(new SolrQuery("*:*")).getResults().getNumFound());
+      assertEquals(
+          "parent doc should be in index2",
+          1,
+          server2.query(new SolrQuery("id:" + parentId)).getResults().getNumFound());
+      assertEquals(
+          "child doc should be in index2",
+          1,
+          server2.query(new SolrQuery("id:" + childId)).getResults().getNumFound());
+      assertEquals(
+          "other doc should be in index1",
+          1,
+          server1.query(new SolrQuery("id:" + otherDocId)).getResults().getNumFound());
+    } finally {
+      h.getCoreContainer().unload("split2");
+      h.getCoreContainer().unload("split1");
+    }
+  }
+
+  /** Utility method to find Ids that hash into which ranges. Uncomment @Test to print. */
+  //  @Test
+  public void printCompositeHashSandbox() {
+    CompositeIdRouter r1 = new CompositeIdRouter();
+    String routeBase = "sea-line!";
+    DocRouter.Range routeBaseRange = r1.keyHashRange(routeBase);
+    List<DocRouter.Range> twoRanges = r1.partitionRange(2, routeBaseRange);
+    System.out.println("splitKeyRange = " + twoRanges);
+
+    // Hash some values and print which range they fall into
+    for (int i = 0; i < 10; i++) {
+      String key = routeBase + i;
+      int hash = r1.sliceHash(key, null, null, null);
+      boolean inA = twoRanges.get(0).includes(hash);
+      boolean inB = twoRanges.get(1).includes(hash);
+
+      // Print which range the key is in
+      System.out.println(key + " in " + (inA ? twoRanges.get(0) : twoRanges.get(1)));
+    }
+  }
+
+  /** Creates a range encompassing the two ids, then splits it in two. Uses PlainIdRouter */
   private List<DocRouter.Range> getRanges(String id1, String id2) {
     // find minHash/maxHash hash ranges
     byte[] bytes = id1.getBytes(StandardCharsets.UTF_8);
     int minHash = Hash.murmurhash3_x86_32(bytes, 0, bytes.length, 0);
     bytes = id2.getBytes(StandardCharsets.UTF_8);
     int maxHash = Hash.murmurhash3_x86_32(bytes, 0, bytes.length, 0);
-
     if (minHash > maxHash) {
       int temp = maxHash;
       maxHash = minHash;
       minHash = temp;
     }
 
+    if (maxHash - minHash < 2) {
+      throw new RuntimeException("The range is too small to split");
+    }
+
     PlainIdRouter router = new PlainIdRouter();
     DocRouter.Range fullRange = new DocRouter.Range(minHash, maxHash);
     return router.partitionRange(2, fullRange);
diff --git a/solr/solr-ref-guide/modules/deployment-guide/pages/collection-management.adoc b/solr/solr-ref-guide/modules/deployment-guide/pages/collection-management.adoc
index f1906e455f7..f4811158ef7 100644
--- a/solr/solr-ref-guide/modules/deployment-guide/pages/collection-management.adoc
+++ b/solr/solr-ref-guide/modules/deployment-guide/pages/collection-management.adoc
@@ -228,6 +228,7 @@ When such a collection is deleted, its autocreated configset will be deleted by
 +
 If this parameter is specified, the router will look at the value of the field in an input document to compute the hash and identify a shard instead of looking at the `uniqueKey` field.
 If the field specified is null in the document, the document will be rejected.
+For nested documents, the route field must match among all the documents in the hierarchy.
 +
 Please note that xref:configuration-guide:realtime-get.adoc[] or retrieval by document ID would also require the parameter `\_route_` (or `shard.keys`) to avoid a distributed search.
 
diff --git a/solr/solr-ref-guide/modules/deployment-guide/pages/node-roles.adoc b/solr/solr-ref-guide/modules/deployment-guide/pages/node-roles.adoc
index a261d23e70c..07dab503bdd 100644
--- a/solr/solr-ref-guide/modules/deployment-guide/pages/node-roles.adoc
+++ b/solr/solr-ref-guide/modules/deployment-guide/pages/node-roles.adoc
@@ -73,7 +73,7 @@ A node with this role can perform duties of an overseer node (unless mode is `di
 
 A node with this role can act as if it has replicas of all collections in the cluster when a query is performed. The workflow is as follows
 
-If the cluster has collections with very large no:of shards, performing distributed requests in your _`data node`_ will lead to
+If the cluster has collections with very large no:of shards, performing distributed requests in your _data node_ will lead to
 
 * large heap utilization
 * frequent GC pauses
diff --git a/solr/solr-ref-guide/modules/deployment-guide/pages/solrcloud-shards-indexing.adoc b/solr/solr-ref-guide/modules/deployment-guide/pages/solrcloud-shards-indexing.adoc
index 21b4027fdfa..d01ce96f3b9 100644
--- a/solr/solr-ref-guide/modules/deployment-guide/pages/solrcloud-shards-indexing.adoc
+++ b/solr/solr-ref-guide/modules/deployment-guide/pages/solrcloud-shards-indexing.adoc
@@ -138,7 +138,7 @@ At query time, you include the prefix(es) along with the number of bits into you
 If you do not want to influence how documents are stored, you don't need to specify a prefix in your document ID.
 
 If you created the collection and defined the "implicit" router at the time of creation, you can additionally define a `router.field` parameter to use a field from each document to identify a shard where the document belongs.
-If the field specified is missing in the document, however, the document will be rejected.
+If the field specified is missing in the document, then the document will be rejected.
 You could also use the `\_route_` parameter to name a specific shard.
 
 == Shard Splitting
diff --git a/solr/solr-ref-guide/modules/indexing-guide/pages/indexing-nested-documents.adoc b/solr/solr-ref-guide/modules/indexing-guide/pages/indexing-nested-documents.adoc
index 939f8f2adc5..fedb60d7e3e 100644
--- a/solr/solr-ref-guide/modules/indexing-guide/pages/indexing-nested-documents.adoc
+++ b/solr/solr-ref-guide/modules/indexing-guide/pages/indexing-nested-documents.adoc
@@ -283,7 +283,7 @@ This makes it much easier to apply
 xref:partial-document-updates.adoc#updating-child-documents[atomic updates] to individual child documents.
 ====
 
-== Maintaining Integrity with Updates and Deletes
+== Maintaining Integrity with Updates, Deletes, and Shard Splits
 
 Nested document trees can be modified with
 xref:partial-document-updates.adoc#atomic-updates[atomic updates] to
@@ -304,6 +304,10 @@ Instead, use delete-by-query (most efficient) or xref:partial-document-updates.a
 If you use Solr's delete-by-query APIs, you *MUST* be careful to ensure that any deletion query is structured to ensure no descendent children remain of any documents that are being deleted.
 *_Doing otherwise will violate integrity assumptions that Solr expects._*
 
+When xref:deployment-guide:shard-management.adoc#splitshard[splitting shards], child documents get transmitted and routed independently.
+If you use the default `router.field`, this is handled correctly for you.
+If your collection has a custom `router.field`, then you must ensure that all documents in a hierarchy have the same value for that field.
+
 == Indexing Anonymous Children
 
 Although not recommended, it is also possible to index child documents "anonymously":

From dbf1c50a67b2e23758ae2381745898f191f0bd47 Mon Sep 17 00:00:00 2001
From: szhou1998 <157182705+szhou1998@users.noreply.github.com>
Date: Tue, 29 Oct 2024 18:38:08 -0700
Subject: [PATCH 09/32] SOLR-16976: Remove log4j-jul jar and use Slf4j bridge
 for JUL (#2703)

Remove log4j-jul jar and use slf4j bridge for JUL to prevent exception from being logged when remote JMX
  is enabled .
Updated jetty.xml to initialize the bridge super early.


Co-authored-by: Eric Pugh <epugh@opensourceconnections.com>
Co-authored-by: David Smiley <dsmiley@apache.org>
---
 solr/CHANGES.txt                         |   3 +
 solr/bin/solr                            |   3 +-
 solr/bin/solr.cmd                        |   2 +-
 solr/licenses/log4j-jul-2.21.0.jar.sha1  |   1 -
 solr/licenses/log4j-jul-LICENSE-ASL.txt  | 201 -----------------------
 solr/licenses/log4j-jul-NOTICE.txt       |  17 --
 solr/packaging/test/test_start_solr.bats |  21 ++-
 solr/server/build.gradle                 |   1 -
 solr/server/etc/jetty.xml                |   2 +
 versions.lock                            |   3 +-
 10 files changed, 24 insertions(+), 230 deletions(-)
 delete mode 100644 solr/licenses/log4j-jul-2.21.0.jar.sha1
 delete mode 100644 solr/licenses/log4j-jul-LICENSE-ASL.txt
 delete mode 100644 solr/licenses/log4j-jul-NOTICE.txt

diff --git a/solr/CHANGES.txt b/solr/CHANGES.txt
index db44d16aa74..e61f29f122a 100644
--- a/solr/CHANGES.txt
+++ b/solr/CHANGES.txt
@@ -191,6 +191,9 @@ Bug Fixes
 
 * SOLR-11191: Splitting shards now routes child-docs with their _root_ field when available so they maintain parent relationship. (Zack Kendall)
 
+* SOLR-16976: Remove log4j-jul jar and use slf4j bridge for JUL to prevent exception from being logged when remote JMX
+  is enabled (Shawn Heisey, Stephen Zhou, Eric Pugh, Christine Poerschke, David Smiley)
+
 Dependency Upgrades
 ---------------------
 (No changes)
diff --git a/solr/bin/solr b/solr/bin/solr
index 812972bf6af..d036a696400 100755
--- a/solr/bin/solr
+++ b/solr/bin/solr
@@ -1030,8 +1030,7 @@ if [ "${SOLR_HOME:0:${#EXAMPLE_DIR}}" = "$EXAMPLE_DIR" ]; then
   SOLR_LOGS_DIR="$SOLR_HOME/../logs"
 fi
 
-# Set the logging manager by default, so that Lucene JUL logs are included with Solr logs.
-LOG4J_CONFIG=("-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager")
+LOG4J_CONFIG=()
 if [ -n "${LOG4J_PROPS:-}" ]; then
   LOG4J_CONFIG+=("-Dlog4j.configurationFile=$LOG4J_PROPS")
 fi
diff --git a/solr/bin/solr.cmd b/solr/bin/solr.cmd
index d6b6f18e248..5a89a12b691 100755
--- a/solr/bin/solr.cmd
+++ b/solr/bin/solr.cmd
@@ -1092,7 +1092,7 @@ IF "%SOLR_SSL_ENABLED%"=="true" (
 set SOLR_LOGS_DIR_QUOTED="%SOLR_LOGS_DIR%"
 set SOLR_DATA_HOME_QUOTED="%SOLR_DATA_HOME%"
 
-set "START_OPTS=%START_OPTS% -Dsolr.log.dir=%SOLR_LOGS_DIR_QUOTED% -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager"
+set "START_OPTS=%START_OPTS% -Dsolr.log.dir=%SOLR_LOGS_DIR_QUOTED%"
 IF NOT "%SOLR_DATA_HOME%"=="" set "START_OPTS=%START_OPTS% -Dsolr.data.home=%SOLR_DATA_HOME_QUOTED%"
 IF NOT DEFINED LOG4J_CONFIG set "LOG4J_CONFIG=%SOLR_SERVER_DIR%\resources\log4j2.xml"
 
diff --git a/solr/licenses/log4j-jul-2.21.0.jar.sha1 b/solr/licenses/log4j-jul-2.21.0.jar.sha1
deleted file mode 100644
index 89f7c940953..00000000000
--- a/solr/licenses/log4j-jul-2.21.0.jar.sha1
+++ /dev/null
@@ -1 +0,0 @@
-f0da61113f4a47654677e6a98b1e13ca7de2483d
diff --git a/solr/licenses/log4j-jul-LICENSE-ASL.txt b/solr/licenses/log4j-jul-LICENSE-ASL.txt
deleted file mode 100644
index f49a4e16e68..00000000000
--- a/solr/licenses/log4j-jul-LICENSE-ASL.txt
+++ /dev/null
@@ -1,201 +0,0 @@
-                                 Apache License
-                           Version 2.0, January 2004
-                        http://www.apache.org/licenses/
-
-   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
-
-   1. Definitions.
-
-      "License" shall mean the terms and conditions for use, reproduction,
-      and distribution as defined by Sections 1 through 9 of this document.
-
-      "Licensor" shall mean the copyright owner or entity authorized by
-      the copyright owner that is granting the License.
-
-      "Legal Entity" shall mean the union of the acting entity and all
-      other entities that control, are controlled by, or are under common
-      control with that entity. For the purposes of this definition,
-      "control" means (i) the power, direct or indirect, to cause the
-      direction or management of such entity, whether by contract or
-      otherwise, or (ii) ownership of fifty percent (50%) or more of the
-      outstanding shares, or (iii) beneficial ownership of such entity.
-
-      "You" (or "Your") shall mean an individual or Legal Entity
-      exercising permissions granted by this License.
-
-      "Source" form shall mean the preferred form for making modifications,
-      including but not limited to software source code, documentation
-      source, and configuration files.
-
-      "Object" form shall mean any form resulting from mechanical
-      transformation or translation of a Source form, including but
-      not limited to compiled object code, generated documentation,
-      and conversions to other media types.
-
-      "Work" shall mean the work of authorship, whether in Source or
-      Object form, made available under the License, as indicated by a
-      copyright notice that is included in or attached to the work
-      (an example is provided in the Appendix below).
-
-      "Derivative Works" shall mean any work, whether in Source or Object
-      form, that is based on (or derived from) the Work and for which the
-      editorial revisions, annotations, elaborations, or other modifications
-      represent, as a whole, an original work of authorship. For the purposes
-      of this License, Derivative Works shall not include works that remain
-      separable from, or merely link (or bind by name) to the interfaces of,
-      the Work and Derivative Works thereof.
-
-      "Contribution" shall mean any work of authorship, including
-      the original version of the Work and any modifications or additions
-      to that Work or Derivative Works thereof, that is intentionally
-      submitted to Licensor for inclusion in the Work by the copyright owner
-      or by an individual or Legal Entity authorized to submit on behalf of
-      the copyright owner. For the purposes of this definition, "submitted"
-      means any form of electronic, verbal, or written communication sent
-      to the Licensor or its representatives, including but not limited to
-      communication on electronic mailing lists, source code control systems,
-      and issue tracking systems that are managed by, or on behalf of, the
-      Licensor for the purpose of discussing and improving the Work, but
-      excluding communication that is conspicuously marked or otherwise
-      designated in writing by the copyright owner as "Not a Contribution."
-
-      "Contributor" shall mean Licensor and any individual or Legal Entity
-      on behalf of whom a Contribution has been received by Licensor and
-      subsequently incorporated within the Work.
-
-   2. Grant of Copyright License. Subject to the terms and conditions of
-      this License, each Contributor hereby grants to You a perpetual,
-      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
-      copyright license to reproduce, prepare Derivative Works of,
-      publicly display, publicly perform, sublicense, and distribute the
-      Work and such Derivative Works in Source or Object form.
-
-   3. Grant of Patent License. Subject to the terms and conditions of
-      this License, each Contributor hereby grants to You a perpetual,
-      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
-      (except as stated in this section) patent license to make, have made,
-      use, offer to sell, sell, import, and otherwise transfer the Work,
-      where such license applies only to those patent claims licensable
-      by such Contributor that are necessarily infringed by their
-      Contribution(s) alone or by combination of their Contribution(s)
-      with the Work to which such Contribution(s) was submitted. If You
-      institute patent litigation against any entity (including a
-      cross-claim or counterclaim in a lawsuit) alleging that the Work
-      or a Contribution incorporated within the Work constitutes direct
-      or contributory patent infringement, then any patent licenses
-      granted to You under this License for that Work shall terminate
-      as of the date such litigation is filed.
-
-   4. Redistribution. You may reproduce and distribute copies of the
-      Work or Derivative Works thereof in any medium, with or without
-      modifications, and in Source or Object form, provided that You
-      meet the following conditions:
-
-      (a) You must give any other recipients of the Work or
-          Derivative Works a copy of this License; and
-
-      (b) You must cause any modified files to carry prominent notices
-          stating that You changed the files; and
-
-      (c) You must retain, in the Source form of any Derivative Works
-          that You distribute, all copyright, patent, trademark, and
-          attribution notices from the Source form of the Work,
-          excluding those notices that do not pertain to any part of
-          the Derivative Works; and
-
-      (d) If the Work includes a "NOTICE" text file as part of its
-          distribution, then any Derivative Works that You distribute must
-          include a readable copy of the attribution notices contained
-          within such NOTICE file, excluding those notices that do not
-          pertain to any part of the Derivative Works, in at least one
-          of the following places: within a NOTICE text file distributed
-          as part of the Derivative Works; within the Source form or
-          documentation, if provided along with the Derivative Works; or,
-          within a display generated by the Derivative Works, if and
-          wherever such third-party notices normally appear. The contents
-          of the NOTICE file are for informational purposes only and
-          do not modify the License. You may add Your own attribution
-          notices within Derivative Works that You distribute, alongside
-          or as an addendum to the NOTICE text from the Work, provided
-          that such additional attribution notices cannot be construed
-          as modifying the License.
-
-      You may add Your own copyright statement to Your modifications and
-      may provide additional or different license terms and conditions
-      for use, reproduction, or distribution of Your modifications, or
-      for any such Derivative Works as a whole, provided Your use,
-      reproduction, and distribution of the Work otherwise complies with
-      the conditions stated in this License.
-
-   5. Submission of Contributions. Unless You explicitly state otherwise,
-      any Contribution intentionally submitted for inclusion in the Work
-      by You to the Licensor shall be under the terms and conditions of
-      this License, without any additional terms or conditions.
-      Notwithstanding the above, nothing herein shall supersede or modify
-      the terms of any separate license agreement you may have executed
-      with Licensor regarding such Contributions.
-
-   6. Trademarks. This License does not grant permission to use the trade
-      names, trademarks, service marks, or product names of the Licensor,
-      except as required for reasonable and customary use in describing the
-      origin of the Work and reproducing the content of the NOTICE file.
-
-   7. Disclaimer of Warranty. Unless required by applicable law or
-      agreed to in writing, Licensor provides the Work (and each
-      Contributor provides its Contributions) on an "AS IS" BASIS,
-      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
-      implied, including, without limitation, any warranties or conditions
-      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
-      PARTICULAR PURPOSE. You are solely responsible for determining the
-      appropriateness of using or redistributing the Work and assume any
-      risks associated with Your exercise of permissions under this License.
-
-   8. Limitation of Liability. In no event and under no legal theory,
-      whether in tort (including negligence), contract, or otherwise,
-      unless required by applicable law (such as deliberate and grossly
-      negligent acts) or agreed to in writing, shall any Contributor be
-      liable to You for damages, including any direct, indirect, special,
-      incidental, or consequential damages of any character arising as a
-      result of this License or out of the use or inability to use the
-      Work (including but not limited to damages for loss of goodwill,
-      work stoppage, computer failure or malfunction, or any and all
-      other commercial damages or losses), even if such Contributor
-      has been advised of the possibility of such damages.
-
-   9. Accepting Warranty or Additional Liability. While redistributing
-      the Work or Derivative Works thereof, You may choose to offer,
-      and charge a fee for, acceptance of support, warranty, indemnity,
-      or other liability obligations and/or rights consistent with this
-      License. However, in accepting such obligations, You may act only
-      on Your own behalf and on Your sole responsibility, not on behalf
-      of any other Contributor, and only if You agree to indemnify,
-      defend, and hold each Contributor harmless for any liability
-      incurred by, or claims asserted against, such Contributor by reason
-      of your accepting any such warranty or additional liability.
-
-   END OF TERMS AND CONDITIONS
-
-   APPENDIX: How to apply the Apache License to your work.
-
-      To apply the Apache License to your work, attach the following
-      boilerplate notice, with the fields enclosed by brackets "[]"
-      replaced with your own identifying information. (Don't include
-      the brackets!)  The text should be enclosed in the appropriate
-      comment syntax for the file format. We also recommend that a
-      file or class name and description of purpose be included on the
-      same "printed page" as the copyright notice for easier
-      identification within third-party archives.
-
-   Copyright [yyyy] [name of copyright owner]
-
-   Licensed under the Apache License, Version 2.0 (the "License");
-   you may not use this file except in compliance with the License.
-   You may obtain a copy of the License at
-
-       http://www.apache.org/licenses/LICENSE-2.0
-
-   Unless required by applicable law or agreed to in writing, software
-   distributed under the License is distributed on an "AS IS" BASIS,
-   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-   See the License for the specific language governing permissions and
-   limitations under the License.
\ No newline at end of file
diff --git a/solr/licenses/log4j-jul-NOTICE.txt b/solr/licenses/log4j-jul-NOTICE.txt
deleted file mode 100644
index ebba5ac0018..00000000000
--- a/solr/licenses/log4j-jul-NOTICE.txt
+++ /dev/null
@@ -1,17 +0,0 @@
-Apache Log4j
-Copyright 1999-2017 Apache Software Foundation
-
-This product includes software developed at
-The Apache Software Foundation (http://www.apache.org/).
-
-ResolverUtil.java
-Copyright 2005-2006 Tim Fennell
-
-Dumbster SMTP test server
-Copyright 2004 Jason Paul Kitchen
-
-TypeUtil.java
-Copyright 2002-2012 Ramnivas Laddad, Juergen Hoeller, Chris Beams
-
-picocli (http://picocli.info)
-Copyright 2017 Remko Popma
\ No newline at end of file
diff --git a/solr/packaging/test/test_start_solr.bats b/solr/packaging/test/test_start_solr.bats
index 546cb9821cb..a4b80efe01d 100644
--- a/solr/packaging/test/test_start_solr.bats
+++ b/solr/packaging/test/test_start_solr.bats
@@ -35,7 +35,7 @@ teardown() {
   solr assert --cloud http://localhost:${SOLR_PORT} --timeout 5000
   solr assert --started http://localhost:${SOLR2_PORT} --timeout 5000
   solr assert --not-cloud http://localhost:${SOLR2_PORT} --timeout 5000
-  
+
   run bash -c 'solr stop --all 2>&1'
   refute_output --partial 'forcefully killing'
 }
@@ -46,7 +46,7 @@ teardown() {
   solr start -p ${SOLR2_PORT}
   solr assert --started http://localhost:${SOLR_PORT} --timeout 5000
   solr assert --started http://localhost:${SOLR2_PORT} --timeout 5000
-  
+
   solr assert --cloud http://localhost:${SOLR_PORT} --timeout 5000
   solr assert --cloud http://localhost:${SOLR2_PORT} --timeout 5000
 
@@ -59,13 +59,24 @@ teardown() {
 @test "check stop command doesn't hang" {
   # for start/stop/restart we parse the args separate from picking the command
   # which means you don't get an error message for passing a start arg, like --jvm-opts to a stop commmand.
-  
+
   # Set a timeout duration (in seconds)
   TIMEOUT_DURATION=2
 
   # make sure that passing a non flag option (i.e --jvm-opts "blah") doesn't hang the stop command.
   run timeout $TIMEOUT_DURATION solr stop --jvm-opts
-    
+
   assert_output --partial "ERROR: JVM options are required when using the --jvm-opts option!"
-  
+
+}
+
+@test "SOLR-16976 solr starts with remote JMX enabled" {
+  export ENABLE_REMOTE_JMX_OPTS=true
+  export RMI_PORT=65535 # need to make sure we don't exceed port range so hard code it
+
+  solr start
+  solr assert --started http://localhost:${SOLR_PORT} --timeout 5000
+
+  run cat ${SOLR_LOGS_DIR}/solr-${SOLR_PORT}-console.log
+  refute_output --partial 'Exception'
 }
diff --git a/solr/server/build.gradle b/solr/server/build.gradle
index d0b722d63f6..93b1196b95b 100644
--- a/solr/server/build.gradle
+++ b/solr/server/build.gradle
@@ -69,7 +69,6 @@ dependencies {
   libExt 'org.apache.logging.log4j:log4j-1.2-api'
   libExt 'org.apache.logging.log4j:log4j-api'
   libExt 'org.apache.logging.log4j:log4j-core'
-  libExt 'org.apache.logging.log4j:log4j-jul'
   libExt 'org.apache.logging.log4j:log4j-layout-template-json'
   libExt 'org.apache.logging.log4j:log4j-slf4j2-impl'
   libExt 'org.apache.logging.log4j:log4j-web'
diff --git a/solr/server/etc/jetty.xml b/solr/server/etc/jetty.xml
index ee3d3f0246f..282de87536f 100644
--- a/solr/server/etc/jetty.xml
+++ b/solr/server/etc/jetty.xml
@@ -47,6 +47,8 @@
     <Set name="detailedDump">false</Set>
   </Get>
 
+  <Call class="org.slf4j.bridge.SLF4JBridgeHandler" name="removeHandlersForRootLogger"/>
+  <Call class="org.slf4j.bridge.SLF4JBridgeHandler" name="install"/>
 
   <!-- =========================================================== -->
   <!-- Http Configuration.                                         -->
diff --git a/versions.lock b/versions.lock
index b7bc1d6f090..f1d2e911ede 100644
--- a/versions.lock
+++ b/versions.lock
@@ -230,9 +230,8 @@ org.apache.kerby:kerby-config:2.0.3 (4 constraints: 593132bc)
 org.apache.kerby:kerby-pkix:2.0.3 (1 constraints: 740b03e5)
 org.apache.kerby:kerby-util:2.0.3 (2 constraints: 6b1837b7)
 org.apache.logging.log4j:log4j-1.2-api:2.21.0 (1 constraints: 3705363b)
-org.apache.logging.log4j:log4j-api:2.21.0 (10 constraints: f07d3453)
+org.apache.logging.log4j:log4j-api:2.21.0 (9 constraints: 976ff7be)
 org.apache.logging.log4j:log4j-core:2.21.0 (5 constraints: 5f510091)
-org.apache.logging.log4j:log4j-jul:2.21.0 (1 constraints: 3705363b)
 org.apache.logging.log4j:log4j-layout-template-json:2.21.0 (1 constraints: 3705363b)
 org.apache.logging.log4j:log4j-slf4j2-impl:2.21.0 (1 constraints: 3705363b)
 org.apache.logging.log4j:log4j-web:2.21.0 (1 constraints: 3705363b)

From 2d9b47ec96f637886552baa500a5632b952d032e Mon Sep 17 00:00:00 2001
From: Sanjay Dutt <sanjaydutt@apache.org>
Date: Wed, 30 Oct 2024 21:08:16 +0530
Subject: [PATCH 10/32] Fix SolrJmxReporterTest#testClosedCore (#2797)

---
 .../reporters/SolrJmxReporterTest.java        | 35 ++++++++++---------
 1 file changed, 18 insertions(+), 17 deletions(-)

diff --git a/solr/core/src/test/org/apache/solr/metrics/reporters/SolrJmxReporterTest.java b/solr/core/src/test/org/apache/solr/metrics/reporters/SolrJmxReporterTest.java
index 4be3a5a583c..3a8f518c53b 100644
--- a/solr/core/src/test/org/apache/solr/metrics/reporters/SolrJmxReporterTest.java
+++ b/solr/core/src/test/org/apache/solr/metrics/reporters/SolrJmxReporterTest.java
@@ -230,28 +230,29 @@ public void testClosedCore() throws Exception {
     assertEquals("Unexpected number of indexDir beans: " + objects.toString(), 1, objects.size());
     final ObjectInstance inst = objects.iterator().next();
     final AtomicBoolean running = new AtomicBoolean(true);
-    try {
-      new Thread(
-              () -> {
-                while (running.get()) {
-                  try {
-                    Object value = TEST_MBEAN_SERVER.getAttribute(inst.getObjectName(), "Value");
-                    assertNotNull(value);
-                  } catch (InstanceNotFoundException x) {
-                    // no longer present
-                    break;
-                  } catch (Exception e) {
-                    fail("Unexpected error retrieving attribute: " + e);
-                  }
+    Thread testThread =
+        new Thread(
+            () -> {
+              while (running.get()) {
+                try {
+                  Object value = TEST_MBEAN_SERVER.getAttribute(inst.getObjectName(), "Value");
+                  assertNotNull(value);
+                } catch (InstanceNotFoundException x) {
+                  // no longer present
+                  break;
+                } catch (Exception e) {
+                  fail("Unexpected error retrieving attribute: " + e);
                 }
-              },
-              "TestMBeanThread")
-          .start();
-
+              }
+            },
+            "TestMBeanThread");
+    try {
+      testThread.start();
       // This should be enough time for the
       Thread.sleep(500);
     } finally {
       running.set(false);
+      testThread.join();
     }
 
     h.getCoreContainer().unload(h.getCore().getName());

From c2091d0258400c9064b8c67fe0d974e367ecccfd Mon Sep 17 00:00:00 2001
From: Christos Malliaridis <malliaridis@apache.org>
Date: Wed, 30 Oct 2024 20:32:45 +0100
Subject: [PATCH 11/32] SOLR-17483: Cleanup zk-related logic in bin/solr.cmd
 (#2816)

* Remove ZK logic from solr.cmd as it is obsolete
* Fix unreachable auth tool in solr.cmd
* Fix invalid references
---
 solr/bin/solr.cmd | 247 +---------------------------------------------
 1 file changed, 2 insertions(+), 245 deletions(-)

diff --git a/solr/bin/solr.cmd b/solr/bin/solr.cmd
index 5a89a12b691..ce6f2aea386 100755
--- a/solr/bin/solr.cmd
+++ b/solr/bin/solr.cmd
@@ -263,7 +263,6 @@ IF "%1"=="assert" goto run_solrcli
 IF "%1"=="zk" goto run_solrcli
 IF "%1"=="export" goto run_solrcli
 IF "%1"=="package" goto run_solrcli
-IF "%1"=="auth" goto run_solrcli
 IF "%1"=="api" goto run_solrcli
 IF "%1"=="post" goto run_solrcli
 
@@ -1011,11 +1010,6 @@ set SCRIPT_SOLR_OPTS=%SCRIPT_SOLR_OPTS% --add-modules jdk.incubator.vector
 IF "%GC_LOG_OPTS%"=="" (
   set GC_LOG_OPTS="-Xlog:gc*"
 )
-if "%JAVA_VENDOR%" == "OpenJ9" (
-  set GC_LOG_OPTS=!GC_LOG_OPTS! "-Xverbosegclog:!SOLR_LOGS_DIR!\solr_gc.log" -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=9 -XX:GCLogFileSize=20M
-) else (
-  set GC_LOG_OPTS=!GC_LOG_OPTS! "-Xloggc:!SOLR_LOGS_DIR!\solr_gc.log" -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=9 -XX:GCLogFileSize=20M
-)
 
 IF "%verbose%"=="1" (
   @echo Starting Solr using the following settings:
@@ -1206,244 +1200,7 @@ SHIFT
 SHIFT
 goto parse_config_args
 
-REM Clumsy to do the state machine thing for -d and -n, but that's required for back-compat
-:parse_zk_args
-IF "%1"=="-V" (
-  goto set_zk_verbose
-) ELSE IF "%1"=="upconfig" (
-  goto set_zk_op
-) ELSE IF "%1"=="downconfig" (
-  goto set_zk_op
-) ELSE IF "%1"=="cp" (
-  goto set_zk_op
-) ELSE IF "%1"=="mv" (
-  goto set_zk_op
-) ELSE IF "%1"=="rm" (
-  goto set_zk_op
-) ELSE IF "%1"=="ls" (
-  goto set_zk_op
-) ELSE IF "%1"=="mkroot" (
-  goto set_zk_op
-) ELSE IF "%1"=="linkconfig" (
-  goto set_zk_op
-) ELSE IF "%1"=="updateacls" (
-  goto set_zk_op
-) ELSE IF "%1"=="-n" (
-  goto set_config_name
-) ELSE IF "%1"=="-r" (
-  goto set_zk_recursive
-) ELSE IF "%1"=="-configname" (
-  goto set_config_name
-) ELSE IF "%1"=="-d" (
-  goto set_configdir
-) ELSE IF "%1"=="-confdir" (
-  goto set_configdir
-) ELSE IF "%1"=="--conf-dir" (
-  goto set_configdir
-) ELSE IF "%1"=="-c" (
-  goto set_collection_zk
-) ELSE IF "%1"=="-z" (
-  goto set_config_zk
-) ELSE IF "!ZK_SRC!"=="" (
-  if not "%~1"=="" (
-    goto set_zk_src
-  )
-) ELSE IF "!ZK_DST!"=="" (
-  IF "%ZK_OP%"=="cp" (
-    goto set_zk_dst
-  )
-  IF "%ZK_OP%"=="mv" (
-    goto set_zk_dst
-  )
-  set ZK_DST="_"
-) ELSE IF NOT "%1"=="" (
-  set ERROR_MSG="Unrecognized or misplaced zk argument %1%"
-  goto zk_short_usage
-)
-goto run_zk
-
-:set_zk_op
-set ZK_OP=%~1
-SHIFT
-goto parse_zk_args
-
-:set_zk_verbose
-set ZK_VERBOSE="--verbose"
-SHIFT
-goto parse_zk_args
-
-:set_config_name
-set CONFIGSET_NAME=%~2
-SHIFT
-SHIFT
-goto parse_zk_args
-
-:set_configdir
-set CONFIGSET_DIR=%~2
-SHIFT
-SHIFT
-goto parse_zk_args
-
-:set_collection_zk
-set ZK_COLLECTION=%~2
-SHIFT
-SHIFT
-goto parse_zk_args
-
-:set_config_zk
-set ZK_HOST=%~2
-SHIFT
-SHIFT
-goto parse_zk_args
-
-:set_zk_src
-set ZK_SRC=%~1
-SHIFT
-goto parse_zk_args
-
-:set_zk_dst
-set ZK_DST=%~1
-SHIFT
-goto parse_zk_args
-
-:set_zk_recursive
-set ZK_RECURSIVE="true"
-SHIFT
-goto parse_zk_args
-
-:run_zk
-IF "!ZK_OP!"=="" (
-  set "ERROR_MSG=Invalid command specified for zk sub-command"
-  goto zk_short_usage
-)
-
-set CONNECTION_PARAMS=""
-
-IF "!ZK_OP!"=="" (
-  set CONNECTION_PARAMS="--solr-url !ZK_SOLR_URL!"
-)
-ELSE (
-  set CONNECTION_PARAMS="--zk-host ZK_HOST!"
-)
-
-IF "!ZK_OP!"=="upconfig" (
-  IF "!CONFIGSET_NAME!"=="" (
-    set ERROR_MSG="-n option must be set for upconfig"
-    goto zk_short_usage
-  )
-  IF "!CONFIGSET_DIR!"=="" (
-    set ERROR_MSG="The -d option must be set for upconfig."
-    goto zk_short_usage
-  )
-  "%JAVA%" %SOLR_SSL_OPTS% %AUTHC_OPTS% %SOLR_ZK_CREDS_AND_ACLS% %SOLR_TOOL_OPTS% -Dsolr.install.dir="%SOLR_TIP%" ^
-  -Dlog4j.configurationFile="file:///%DEFAULT_SERVER_DIR%\resources\log4j2-console.xml" ^
-  -classpath "%DEFAULT_SERVER_DIR%\solr-webapp\webapp\WEB-INF\lib\*;%DEFAULT_SERVER_DIR%\lib\ext\*" ^
-  org.apache.solr.cli.SolrCLI !ZK_OP! --conf-name !CONFIGSET_NAME! --conf-dir !CONFIGSET_DIR! %CONNECTION_PARAMS% %ZK_VERBOSE%^
-) ELSE IF "!ZK_OP!"=="downconfig" (
-  IF "!CONFIGSET_NAME!"=="" (
-    set ERROR_MSG="-n option must be set for downconfig"
-    goto zk_short_usage
-  )
-  IF "!CONFIGSET_DIR!"=="" (
-    set ERROR_MSG="The -d option must be set for downconfig."
-    goto zk_short_usage
-  )
-  "%JAVA%" %SOLR_SSL_OPTS% %AUTHC_OPTS% %SOLR_ZK_CREDS_AND_ACLS% %SOLR_TOOL_OPTS% -Dsolr.install.dir="%SOLR_TIP%" ^
-  -Dlog4j.configurationFile="file:///%DEFAULT_SERVER_DIR%\resources\log4j2-console.xml" ^
-  -classpath "%DEFAULT_SERVER_DIR%\solr-webapp\webapp\WEB-INF\lib\*;%DEFAULT_SERVER_DIR%\lib\ext\*" ^
-  org.apache.solr.cli.SolrCLI !ZK_OP! --conf-name !CONFIGSET_NAME! --conf-dir !CONFIGSET_DIR! -z !ZK_HOST! %ZK_VERBOSE%
-) ELSE IF "!ZK_OP!"=="linkconfig" (
-  IF "!CONFIGSET_NAME!"=="" (
-    set ERROR_MSG="-n option must be set for linkconfig"
-    goto zk_short_usage
-  )
-  IF "!ZK_COLLECTION!"=="" (
-    set ERROR_MSG="The -c option must be set for linkconfig."
-    goto zk_short_usage
-  )
-  "%JAVA%" %SOLR_SSL_OPTS% %AUTHC_OPTS% %SOLR_ZK_CREDS_AND_ACLS% %SOLR_TOOL_OPTS% -Dsolr.install.dir="%SOLR_TIP%" ^
-  -Dlog4j.configurationFile="file:///%DEFAULT_SERVER_DIR%\resources\log4j2-console.xml" ^
-  -classpath "%DEFAULT_SERVER_DIR%\solr-webapp\webapp\WEB-INF\lib\*;%DEFAULT_SERVER_DIR%\lib\ext\*" ^
-  org.apache.solr.cli.SolrCLI !ZK_OP! --conf-name !CONFIGSET_NAME! -c !ZK_COLLECTION! -z !ZK_HOST! %ZK_VERBOSE%
-) ELSE IF "!ZK_OP!"=="updateacls" (
-  IF "%ZK_SRC"=="" (
-    set ERROR_MSG="Zookeeper path to remove must be specified when using the 'ls' command"
-    goto zk_short_usage
-  )
-  "%JAVA%" %SOLR_SSL_OPTS% %AUTHC_OPTS% %SOLR_ZK_CREDS_AND_ACLS% %SOLR_TOOL_OPTS% -Dsolr.install.dir="%SOLR_TIP%" ^
-  -Dlog4j.configurationFile="file:///%DEFAULT_SERVER_DIR%\resources\log4j2-console.xml" ^
-  -classpath "%DEFAULT_SERVER_DIR%\solr-webapp\webapp\WEB-INF\lib\*;%DEFAULT_SERVER_DIR%\lib\ext\*" ^
-  org.apache.solr.cli.SolrCLI !ZK_OP! --path !ZK_SRC! -z !ZK_HOST! %ZK_VERBOSE%
-) ELSE IF "!ZK_OP!"=="cp" (
-  IF "%ZK_SRC%"=="" (
-    set ERROR_MSG="<src> must be specified for 'cp' command"
-    goto zk_short_usage
-  )
-  IF "%ZK_DST%"=="" (
-    set ERROR_MSG=<dest> must be specified for 'cp' command"
-    goto zk_short_usage
-  )
-  IF NOT "!ZK_SRC:~0,3!"=="zk:" (
-    IF NOT "!%ZK_DST:~0,3!"=="zk:" (
-      set ERROR_MSG="At least one of src or dst must be prefixed by 'zk:'"
-      goto zk_short_usage
-  )
-  )
-  "%JAVA%" %SOLR_SSL_OPTS% %AUTHC_OPTS% %SOLR_ZK_CREDS_AND_ACLS% %SOLR_TOOL_OPTS% -Dsolr.install.dir="%SOLR_TIP%" ^
-  -Dlog4j.configurationFile="file:///%DEFAULT_SERVER_DIR%\resources\log4j2-console.xml" ^
-  -classpath "%DEFAULT_SERVER_DIR%\solr-webapp\webapp\WEB-INF\lib\*;%DEFAULT_SERVER_DIR%\lib\ext\*" ^
-  org.apache.solr.cli.SolrCLI !ZK_OP! -z !ZK_HOST! --source !ZK_SRC! --destination !ZK_DST! --recursive !ZK_RECURSIVE! %ZK_VERBOSE%
-) ELSE IF "!ZK_OP!"=="mv" (
-  IF "%ZK_SRC%"=="" (
-    set ERROR_MSG="<src> must be specified for 'mv' command"
-    goto zk_short_usage
-  )
-  IF "%ZK_DST%"=="" (
-    set ERROR_MSG="<dest> must be specified for 'mv' command"
-    goto zk_short_usage
-  )
-  "%JAVA%" %SOLR_SSL_OPTS% %AUTHC_OPTS% %SOLR_ZK_CREDS_AND_ACLS% %SOLR_TOOL_OPTS% -Dsolr.install.dir="%SOLR_TIP%" ^
-  -Dlog4j.configurationFile="file:///%DEFAULT_SERVER_DIR%\resources\log4j2-console.xml" ^
-  -classpath "%DEFAULT_SERVER_DIR%\solr-webapp\webapp\WEB-INF\lib\*;%DEFAULT_SERVER_DIR%\lib\ext\*" ^
-  org.apache.solr.cli.SolrCLI !ZK_OP! -z !ZK_HOST! --source !ZK_SRC! --destination !ZK_DST! %ZK_VERBOSE%
-) ELSE IF "!ZK_OP!"=="rm" (
-  IF "%ZK_SRC"=="" (
-    set ERROR_MSG="Zookeeper path to remove must be specified when using the 'rm' command"
-    goto zk_short_usage
-  )
-  "%JAVA%" %SOLR_SSL_OPTS% %AUTHC_OPTS% %SOLR_ZK_CREDS_AND_ACLS% %SOLR_TOOL_OPTS% -Dsolr.install.dir="%SOLR_TIP%" ^
-  -Dlog4j.configurationFile="file:///%DEFAULT_SERVER_DIR%\resources\log4j2-console.xml" ^
-  -classpath "%DEFAULT_SERVER_DIR%\solr-webapp\webapp\WEB-INF\lib\*;%DEFAULT_SERVER_DIR%\lib\ext\*" ^
-  org.apache.solr.cli.SolrCLI !ZK_OP! -z !ZK_HOST! --path !ZK_SRC! --recursive !ZK_RECURSIVE! %ZK_VERBOSE%
-) ELSE IF "!ZK_OP!"=="ls" (
-  IF "%ZK_SRC"=="" (
-    set ERROR_MSG="Zookeeper path to remove must be specified when using the 'ls' command"
-    goto zk_short_usage
-  )
-  "%JAVA%" %SOLR_SSL_OPTS% %AUTHC_OPTS% %SOLR_ZK_CREDS_AND_ACLS% %SOLR_TOOL_OPTS% -Dsolr.install.dir="%SOLR_TIP%" ^
-  -Dlog4j.configurationFile="file:///%DEFAULT_SERVER_DIR%\resources\log4j2-console.xml" ^
-  -classpath "%DEFAULT_SERVER_DIR%\solr-webapp\webapp\WEB-INF\lib\*;%DEFAULT_SERVER_DIR%\lib\ext\*" ^
-  org.apache.solr.cli.SolrCLI !ZK_OP! -z !ZK_HOST! --path !ZK_SRC! --recursive !ZK_RECURSIVE! %ZK_VERBOSE%
-) ELSE IF "!ZK_OP!"=="mkroot" (
-  IF "%ZK_SRC"=="" (
-    set ERROR_MSG="Zookeeper path to create must be specified when using the 'mkroot' command"
-    goto zk_short_usage
-  )
-  "%JAVA%" %SOLR_SSL_OPTS% %AUTHC_OPTS% %SOLR_ZK_CREDS_AND_ACLS% %SOLR_TOOL_OPTS% -Dsolr.install.dir="%SOLR_TIP%" ^
-  -Dlog4j.configurationFile="file:///%SOLR_SERVER_DIR%\resources\log4j2-console.xml" ^
-  -classpath "%DEFAULT_SERVER_DIR%\solr-webapp\webapp\WEB-INF\lib\*;%DEFAULT_SERVER_DIR%\lib\ext\*" ^
-  org.apache.solr.cli.SolrCLI !ZK_OP! -z !ZK_HOST! --path !ZK_SRC! %ZK_VERBOSE%
-) ELSE (
-  set ERROR_MSG="Unknown zk option !ZK_OP!"
-  goto zk_short_usage
-)
-goto done
-
-
 :run_auth
-IF "%1"=="-help" goto usage
-IF "%1"=="-usage" goto usage
-
 REM Options parsing.
 REM Note: With the following technique of parsing, it is not possible
 REM       to have an option without a value.
@@ -1521,9 +1278,9 @@ IF "%FIRST_ARG%"=="start" (
 ) ELSE IF "%FIRST_ARG%"=="create" (
   goto run_solrcli
 ) ELSE IF "%FIRST_ARG%"=="zk" (
-  goto zk_short_usage
+  goto run_solrcli
 ) ELSE IF "%FIRST_ARG%"=="auth" (
-  goto auth_usage
+  goto run_solrcli
 ) ELSE IF "%FIRST_ARG%"=="status" (
   goto run_solrcli
 )

From e5d15cc84f8ad790e1b6c901d74bd86b7c1de17c Mon Sep 17 00:00:00 2001
From: Houston Putman <houston@apache.org>
Date: Wed, 30 Oct 2024 16:55:22 -0500
Subject: [PATCH 12/32] SOLR-16116: Use apache curator to manage the Solr
 Zookeeper interactions (#760)

Co-authored-by: Kevin Risden <krisden@apache.org>
---
 .../randomization/policies/solr-tests.policy  |   2 +
 solr/CHANGES.txt                              |   3 +
 solr/core/build.gradle                        |  11 +-
 .../org/apache/solr/cloud/LeaderElector.java  |   7 +-
 .../java/org/apache/solr/cloud/Overseer.java  |  20 +-
 .../solr/cloud/OverseerTaskProcessor.java     |  11 +-
 .../cloud/ShardLeaderElectionContextBase.java |  54 +-
 .../org/apache/solr/cloud/ZkController.java   |  67 +-
 .../apache/solr/cloud/ZkDistributedQueue.java |  31 +-
 .../org/apache/solr/cloud/ZkShardTerms.java   |  15 +-
 .../org/apache/solr/core/ZkContainer.java     |   3 +-
 .../solr/handler/component/SearchHandler.java |   4 +-
 .../DistributedZkUpdateProcessor.java         |   2 +-
 .../apache/solr/cli/ZkSubcommandsTest.java    |  10 +-
 .../solr/cloud/ConnectionManagerTest.java     | 186 -----
 .../solr/cloud/DistributedQueueTest.java      |  23 +-
 .../apache/solr/cloud/LeaderElectionTest.java |  72 +-
 .../solr/cloud/MultiThreadedOCPTest.java      |   3 +-
 ...OfBoxZkACLAndCredentialsProvidersTest.java |   2 +-
 ...iddenZkACLAndCredentialsProvidersTest.java | 174 +++--
 .../org/apache/solr/cloud/OverseerTest.java   |  11 +-
 .../cloud/TestLeaderElectionZkExpiry.java     |   3 +-
 .../cloud/TestPullReplicaErrorHandling.java   |   2 +
 .../cloud/TestSizeLimitedDistributedMap.java  |   3 +-
 .../apache/solr/cloud/ZkSolrClientTest.java   |  63 +-
 .../cloud/overseer/ZkStateWriterTest.java     |   9 +-
 ...ICENSE-ASL.txt => curator-LICENSE-ASL.txt} |   0
 ...r-client-NOTICE.txt => curator-NOTICE.txt} |   0
 solr/licenses/curator-client-5.7.0.jar.sha1   |   1 -
 solr/licenses/curator-client-5.7.1.jar.sha1   |   1 +
 .../licenses/curator-framework-5.7.0.jar.sha1 |   1 -
 .../licenses/curator-framework-5.7.1.jar.sha1 |   1 +
 .../curator-framework-LICENSE-ASL.txt         | 202 ------
 solr/licenses/curator-framework-NOTICE.txt    |   5 -
 solr/licenses/curator-recipes-5.7.0.jar.sha1  |   1 -
 solr/licenses/curator-recipes-5.7.1.jar.sha1  |   1 +
 solr/licenses/curator-recipes-NOTICE.txt      |   5 -
 solr/licenses/curator-test-5.7.1.jar.sha1     |   1 +
 .../licenses/junit-jupiter-api-5.6.2.jar.sha1 |   1 +
 .../junit-platform-commons-1.6.2.jar.sha1     |   1 +
 solr/licenses/opentest4j-1.2.0.jar.sha1       |   1 +
 ...NSE-ASL.txt => opentest4j-LICENSE-ASL.txt} |   8 +-
 solr/licenses/opentest4j-NOTICE.txt           |  13 +
 solr/modules/hadoop-auth/build.gradle         |  19 +-
 .../hadoop/DelegationTokenKerberosFilter.java | 129 +---
 .../security/hadoop/HadoopAuthFilter.java     | 126 +---
 .../security/hadoop/HadoopAuthPlugin.java     |   2 +
 .../solr/security/hadoop/KerberosPlugin.java  |   3 +
 .../hadoop/SaslZkACLProviderTest.java         |   8 +-
 .../TestImpersonationWithHadoopAuth.java      |   3 +-
 .../TestSolrCloudWithDelegationTokens.java    |   9 +-
 solr/solrj-zookeeper/build.gradle             |   7 +
 .../solrj/cloud/DistribStateManager.java      |   6 +-
 .../solrj/impl/ZkDistribStateManager.java     |   7 +-
 .../solr/common/cloud/BeforeReconnect.java    |  13 +-
 .../solr/common/cloud/ConnectionManager.java  | 318 ---------
 .../cloud/DefaultConnectionStrategy.java      |  74 --
 .../common/cloud/DefaultZkACLProvider.java    |  32 +-
 .../cloud/DefaultZkCredentialsProvider.java   |  21 +-
 .../cloud/DigestZkCredentialsProvider.java    |  17 +-
 .../solr/common/cloud/OnDisconnect.java}      |  25 +-
 .../apache/solr/common/cloud/OnReconnect.java |  22 +-
 .../common/cloud/PerReplicaStatesOps.java     |  10 +-
 .../solr/common/cloud/SaslZkACLProvider.java  |   4 +-
 .../cloud/SecurityAwareZkACLProvider.java     |  52 +-
 .../solr/common/cloud/SolrZkClient.java       | 666 ++++++++++--------
 ...redentialsDigestZkCredentialsProvider.java |   5 +-
 .../solr/common/cloud/ZkACLProvider.java      |   8 +-
 .../cloud/ZkClientConnectionStrategy.java     | 155 ----
 .../solr/common/cloud/ZkCmdExecutor.java      | 103 ---
 .../common/cloud/ZkCredentialsProvider.java   |  25 +-
 .../solr/common/cloud/ZkStateReader.java      |  10 +-
 .../solr/common/cloud/SolrZkClientTest.java   |  39 +-
 .../common/cloud/TestZkConfigSetService.java  |  90 +--
 .../solrj/impl/SolrZkClientTimeout.java       |   2 +
 .../PerReplicaStatesIntegrationTest.java      |   1 +
 .../cloud/SolrZkClientCompressedDataTest.java |   5 +-
 solr/test-framework/build.gradle              |  12 +
 .../apache/solr/SolrIgnoredThreadsFilter.java |   5 +
 .../java/org/apache/solr/SolrTestCaseJ4.java  |   2 -
 ...tZkACLAndCredentialsProvidersTestBase.java |   6 +-
 .../org/apache/solr/cloud/ChaosMonkey.java    |  23 +-
 .../solr/cloud/MiniSolrCloudCluster.java      |  10 +-
 .../org/apache/solr/cloud/ZkTestServer.java   |  57 +-
 versions.lock                                 |  21 +-
 versions.props                                |   2 +-
 86 files changed, 1025 insertions(+), 2163 deletions(-)
 delete mode 100644 solr/core/src/test/org/apache/solr/cloud/ConnectionManagerTest.java
 rename solr/licenses/{curator-client-LICENSE-ASL.txt => curator-LICENSE-ASL.txt} (100%)
 rename solr/licenses/{curator-client-NOTICE.txt => curator-NOTICE.txt} (100%)
 delete mode 100644 solr/licenses/curator-client-5.7.0.jar.sha1
 create mode 100644 solr/licenses/curator-client-5.7.1.jar.sha1
 delete mode 100644 solr/licenses/curator-framework-5.7.0.jar.sha1
 create mode 100644 solr/licenses/curator-framework-5.7.1.jar.sha1
 delete mode 100644 solr/licenses/curator-framework-LICENSE-ASL.txt
 delete mode 100644 solr/licenses/curator-framework-NOTICE.txt
 delete mode 100644 solr/licenses/curator-recipes-5.7.0.jar.sha1
 create mode 100644 solr/licenses/curator-recipes-5.7.1.jar.sha1
 delete mode 100644 solr/licenses/curator-recipes-NOTICE.txt
 create mode 100644 solr/licenses/curator-test-5.7.1.jar.sha1
 create mode 100644 solr/licenses/junit-jupiter-api-5.6.2.jar.sha1
 create mode 100644 solr/licenses/junit-platform-commons-1.6.2.jar.sha1
 create mode 100644 solr/licenses/opentest4j-1.2.0.jar.sha1
 rename solr/licenses/{curator-recipes-LICENSE-ASL.txt => opentest4j-LICENSE-ASL.txt} (99%)
 create mode 100644 solr/licenses/opentest4j-NOTICE.txt
 delete mode 100644 solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/ConnectionManager.java
 delete mode 100644 solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/DefaultConnectionStrategy.java
 rename solr/{test-framework/src/java/org/apache/solr/cloud/TestConnectionStrategy.java => solrj-zookeeper/src/java/org/apache/solr/common/cloud/OnDisconnect.java} (54%)
 delete mode 100644 solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/ZkClientConnectionStrategy.java
 delete mode 100644 solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/ZkCmdExecutor.java

diff --git a/gradle/testing/randomization/policies/solr-tests.policy b/gradle/testing/randomization/policies/solr-tests.policy
index 8183cf1e1e7..61df0871a35 100644
--- a/gradle/testing/randomization/policies/solr-tests.policy
+++ b/gradle/testing/randomization/policies/solr-tests.policy
@@ -50,6 +50,8 @@ grant {
   permission java.net.SocketPermission "127.0.0.1:4", "connect,resolve";
   permission java.net.SocketPermission "127.0.0.1:6", "connect,resolve";
   permission java.net.SocketPermission "127.0.0.1:8", "connect,resolve";
+  // Used as an invalid ZK host
+  permission java.net.SocketPermission "----------:33332", "connect,resolve";
 
   // Basic permissions needed for Lucene to work:
   permission java.util.PropertyPermission "*", "read,write";
diff --git a/solr/CHANGES.txt b/solr/CHANGES.txt
index e61f29f122a..83327d8605b 100644
--- a/solr/CHANGES.txt
+++ b/solr/CHANGES.txt
@@ -23,6 +23,9 @@ Improvements
 
 * SOLR-17077: When a shard rejoins leader election, leave previous election only once to save unneeded calls to Zookeeper. (Pierre Salagnac)
 
+* SOLR-16116: Apache Curator is now used to manage all Solr Zookeeper interactions. This should provide more stability in the Solr-Zookeeper interactions.
+  The solrj-zookeeper module, now has a dependency on curator. (Houston Putman, Kevin Risden, Mike Drob, David Smiley)
+
 Optimizations
 ---------------------
 (No changes)
diff --git a/solr/core/build.gradle b/solr/core/build.gradle
index b36843fa568..92bbfbe1120 100644
--- a/solr/core/build.gradle
+++ b/solr/core/build.gradle
@@ -49,7 +49,6 @@ dependencies {
   api project(':solr:solrj-zookeeper')
   api project(':solr:solrj-streaming')
 
-
   api 'io.dropwizard.metrics:metrics-core'
   implementation ('io.dropwizard.metrics:metrics-graphite', {
     exclude group: "com.rabbitmq", module: "amqp-client"
@@ -125,6 +124,16 @@ dependencies {
   implementation 'org.eclipse.jetty.toolchain:jetty-servlet-api'
 
   // ZooKeeper
+
+  implementation('org.apache.curator:curator-framework', {
+    exclude group: 'org.apache.zookeeper', module: 'zookeeper'
+  })
+  implementation('org.apache.curator:curator-client', {
+    exclude group: 'org.apache.zookeeper', module: 'zookeeper'
+  })
+  testImplementation('org.apache.curator:curator-test', {
+    exclude group: 'org.apache.zookeeper', module: 'zookeeper'
+  })
   implementation('org.apache.zookeeper:zookeeper', {
     exclude group: "org.apache.yetus", module: "audience-annotations"
   })
diff --git a/solr/core/src/java/org/apache/solr/cloud/LeaderElector.java b/solr/core/src/java/org/apache/solr/cloud/LeaderElector.java
index 544437c5d67..f058aefe295 100644
--- a/solr/core/src/java/org/apache/solr/cloud/LeaderElector.java
+++ b/solr/core/src/java/org/apache/solr/cloud/LeaderElector.java
@@ -25,7 +25,6 @@
 import java.util.regex.Matcher;
 import java.util.regex.Pattern;
 import org.apache.solr.cloud.ZkController.ContextKey;
-import org.apache.solr.common.AlreadyClosedException;
 import org.apache.solr.common.SolrException;
 import org.apache.solr.common.cloud.SolrZkClient;
 import org.apache.solr.common.cloud.ZkMaintenanceUtils;
@@ -107,7 +106,7 @@ private void checkIfIamLeader(final ElectionContext context, boolean replacement
 
     // If any double-registrations exist for me, remove all but this latest one!
     // TODO: can we even get into this state?
-    String prefix = zkClient.getZooKeeper().getSessionId() + "-" + context.id + "-";
+    String prefix = zkClient.getZkSessionId() + "-" + context.id + "-";
     Iterator<String> it = seqs.iterator();
     while (it.hasNext()) {
       String elec = it.next();
@@ -227,7 +226,7 @@ public int joinElection(ElectionContext context, boolean replacement, boolean jo
 
     final String shardsElectZkPath = context.electionPath + LeaderElector.ELECTION_NODE;
 
-    long sessionId = zkClient.getZooKeeper().getSessionId();
+    long sessionId = zkClient.getZkSessionId();
     String id = sessionId + "-" + context.id;
     String leaderSeqPath = null;
     boolean cont = true;
@@ -348,7 +347,7 @@ public void process(WatchedEvent event) {
       try {
         // am I the next leader?
         checkIfIamLeader(context, true);
-      } catch (AlreadyClosedException e) {
+      } catch (IllegalStateException e) {
 
       } catch (Exception e) {
         if (!zkClient.isClosed()) {
diff --git a/solr/core/src/java/org/apache/solr/cloud/Overseer.java b/solr/core/src/java/org/apache/solr/cloud/Overseer.java
index bd91a226a5d..378b2ff7043 100644
--- a/solr/core/src/java/org/apache/solr/cloud/Overseer.java
+++ b/solr/core/src/java/org/apache/solr/cloud/Overseer.java
@@ -316,7 +316,7 @@ public void run() {
               // the workQueue is empty now, use stateUpdateQueue as fallback queue
               fallbackQueue = stateUpdateQueue;
               fallbackQueueSize = 0;
-            } catch (AlreadyClosedException e) {
+            } catch (IllegalStateException e) {
               return;
             } catch (KeeperException.SessionExpiredException e) {
               log.warn("Solr cannot talk to ZK, exiting Overseer work queue loop", e);
@@ -342,7 +342,7 @@ public void run() {
           } catch (InterruptedException e) {
             Thread.currentThread().interrupt();
             return;
-          } catch (AlreadyClosedException e) {
+          } catch (IllegalStateException e) {
 
           } catch (Exception e) {
             log.error("Exception in Overseer main queue loop", e);
@@ -402,7 +402,7 @@ public void run() {
           } catch (InterruptedException e) {
             Thread.currentThread().interrupt();
             return;
-          } catch (AlreadyClosedException e) {
+          } catch (IllegalStateException e) {
 
           } catch (Exception e) {
             log.error("Exception in Overseer main queue loop", e);
@@ -414,7 +414,9 @@ public void run() {
           log.info("Overseer Loop exiting : {}", LeaderElector.getNodeName(myId));
         }
         // do this in a separate thread because any wait is interrupted in this main thread
-        new Thread(this::checkIfIamStillLeader, "OverseerExitThread").start();
+        Thread checkLeaderThread = new Thread(this::checkIfIamStillLeader, "OverseerExitThread");
+        checkLeaderThread.setDaemon(true);
+        checkLeaderThread.start();
       }
     }
 
@@ -480,7 +482,7 @@ private void checkIfIamStillLeader() {
       byte[] data;
       try {
         data = zkClient.getData(path, null, stat, true);
-      } catch (AlreadyClosedException e) {
+      } catch (IllegalStateException e) {
         return;
       } catch (Exception e) {
         log.warn("Error communicating with ZooKeeper", e);
@@ -634,7 +636,7 @@ private LeaderStatus amILeader() {
       } catch (InterruptedException e) {
         success = false;
         Thread.currentThread().interrupt();
-      } catch (AlreadyClosedException e) {
+      } catch (IllegalStateException e) {
         success = false;
       } catch (Exception e) {
         success = false;
@@ -1047,11 +1049,7 @@ ZkDistributedQueue getOverseerQuitNotificationQueue() {
    */
   ZkDistributedQueue getStateUpdateQueue(Stats zkStats) {
     return new ZkDistributedQueue(
-        reader.getZkClient(),
-        "/overseer/queue",
-        zkStats,
-        STATE_UPDATE_MAX_QUEUE,
-        () -> Overseer.this.isClosed() || zkController.getCoreContainer().isShutDown());
+        reader.getZkClient(), "/overseer/queue", zkStats, STATE_UPDATE_MAX_QUEUE);
   }
 
   /**
diff --git a/solr/core/src/java/org/apache/solr/cloud/OverseerTaskProcessor.java b/solr/core/src/java/org/apache/solr/cloud/OverseerTaskProcessor.java
index eb5661885c4..b63afbcf0c4 100644
--- a/solr/core/src/java/org/apache/solr/cloud/OverseerTaskProcessor.java
+++ b/solr/core/src/java/org/apache/solr/cloud/OverseerTaskProcessor.java
@@ -36,7 +36,6 @@
 import java.util.function.Predicate;
 import org.apache.solr.cloud.Overseer.LeaderStatus;
 import org.apache.solr.cloud.OverseerTaskQueue.QueueEvent;
-import org.apache.solr.common.AlreadyClosedException;
 import org.apache.solr.common.cloud.SolrZkClient;
 import org.apache.solr.common.cloud.ZkNodeProps;
 import org.apache.solr.common.cloud.ZkStateReader;
@@ -188,7 +187,7 @@ public void run() {
       // We don't need to handle this. This is just a fail-safe which comes in handy in skipping
       // already processed async calls.
       log.error("KeeperException", e);
-    } catch (AlreadyClosedException e) {
+    } catch (IllegalStateException ignore) {
       return;
     } catch (InterruptedException e) {
       Thread.currentThread().interrupt();
@@ -202,7 +201,7 @@ public void run() {
 
     try {
       prioritizer.prioritizeOverseerNodes(myId);
-    } catch (AlreadyClosedException e) {
+    } catch (IllegalStateException ignore) {
       return;
     } catch (Exception e) {
       if (!zkStateReader.getZkClient().isClosed()) {
@@ -395,7 +394,7 @@ public void run() {
         } catch (InterruptedException e) {
           Thread.currentThread().interrupt();
           return;
-        } catch (AlreadyClosedException ignore) {
+        } catch (IllegalStateException ignore) {
 
         } catch (Exception e) {
           log.error("Exception processing", e);
@@ -602,7 +601,7 @@ public void run() {
               response.getResponse());
         }
         success = true;
-      } catch (AlreadyClosedException ignore) {
+      } catch (IllegalStateException ignore) {
 
       } catch (KeeperException e) {
         log.error("KeeperException", e);
@@ -617,7 +616,7 @@ public void run() {
           // Reset task from tracking data structures so that it can be retried.
           try {
             resetTaskWithException(messageHandler, head.getId(), asyncId, taskKey, message);
-          } catch (AlreadyClosedException ignore) {
+          } catch (IllegalStateException ignore) {
 
           }
         }
diff --git a/solr/core/src/java/org/apache/solr/cloud/ShardLeaderElectionContextBase.java b/solr/core/src/java/org/apache/solr/cloud/ShardLeaderElectionContextBase.java
index 6d97d785b0e..7f4548b6912 100644
--- a/solr/core/src/java/org/apache/solr/cloud/ShardLeaderElectionContextBase.java
+++ b/solr/core/src/java/org/apache/solr/cloud/ShardLeaderElectionContextBase.java
@@ -19,6 +19,8 @@
 
 import java.lang.invoke.MethodHandles;
 import java.util.List;
+import org.apache.curator.framework.api.transaction.CuratorTransactionResult;
+import org.apache.curator.framework.api.transaction.OperationType;
 import org.apache.solr.cloud.overseer.OverseerAction;
 import org.apache.solr.common.SolrException;
 import org.apache.solr.common.SolrException.ErrorCode;
@@ -36,11 +38,6 @@
 import org.apache.zookeeper.KeeperException;
 import org.apache.zookeeper.KeeperException.NoNodeException;
 import org.apache.zookeeper.KeeperException.NodeExistsException;
-import org.apache.zookeeper.Op;
-import org.apache.zookeeper.OpResult;
-import org.apache.zookeeper.OpResult.SetDataResult;
-import org.apache.zookeeper.ZooDefs;
-import org.apache.zookeeper.data.Stat;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
@@ -106,10 +103,11 @@ public void cancelElection() throws InterruptedException, KeeperException {
               leaderPath,
               leaderZkNodeParentVersion);
           String parent = ZkMaintenanceUtils.getZkParent(leaderPath);
-          List<Op> ops =
-              List.of(Op.check(parent, leaderZkNodeParentVersion), Op.delete(leaderPath, -1));
-          zkClient.multi(ops, true);
+          zkClient.multi(
+              op -> op.check().withVersion(leaderZkNodeParentVersion).forPath(parent),
+              op -> op.delete().withVersion(-1).forPath(leaderPath));
         } catch (InterruptedException e) {
+          Thread.currentThread().interrupt();
           throw e;
         } catch (IllegalArgumentException e) {
           log.error("Illegal argument", e);
@@ -144,27 +142,25 @@ void runLeaderProcess(boolean weAreReplacement, int pauseBeforeStartMs)
               // be used to make sure we only remove our own leader registration node.
               // The setData call used to get the parent version is also the trigger to
               // increment the version. We also do a sanity check that our leaderSeqPath exists.
-              List<Op> ops =
-                  List.of(
-                      Op.check(leaderSeqPath, -1),
-                      Op.create(
-                          leaderPath,
-                          Utils.toJSON(leaderProps),
-                          zkClient.getZkACLProvider().getACLsToAdd(leaderPath),
-                          CreateMode.EPHEMERAL),
-                      Op.setData(parent, null, -1));
-              List<OpResult> results;
-
-              results = zkClient.multi(ops, true);
-              for (OpResult result : results) {
-                if (result.getType() == ZooDefs.OpCode.setData) {
-                  SetDataResult dresult = (SetDataResult) result;
-                  Stat stat = dresult.getStat();
-                  leaderZkNodeParentVersion = stat.getVersion();
-                  return;
-                }
-              }
-              assert leaderZkNodeParentVersion != null;
+              List<CuratorTransactionResult> results =
+                  zkClient.multi(
+                      op -> op.check().withVersion(-1).forPath(leaderSeqPath),
+                      op ->
+                          op.create()
+                              .withMode(CreateMode.EPHEMERAL)
+                              .forPath(leaderPath, Utils.toJSON(leaderProps)),
+                      op -> op.setData().withVersion(-1).forPath(parent, null));
+              leaderZkNodeParentVersion =
+                  results.stream()
+                      .filter(
+                          CuratorTransactionResult.ofTypeAndPath(OperationType.SET_DATA, parent))
+                      .findFirst()
+                      .orElseThrow(
+                          () ->
+                              new RuntimeException(
+                                  "Could not set data for parent path in ZK: " + parent))
+                      .getResultStat()
+                      .getVersion();
             }
           });
     } catch (NoNodeException e) {
diff --git a/solr/core/src/java/org/apache/solr/cloud/ZkController.java b/solr/core/src/java/org/apache/solr/cloud/ZkController.java
index 33732ea0a94..59faced1b84 100644
--- a/solr/core/src/java/org/apache/solr/cloud/ZkController.java
+++ b/solr/core/src/java/org/apache/solr/cloud/ZkController.java
@@ -52,6 +52,7 @@
 import java.util.concurrent.atomic.AtomicReference;
 import java.util.function.Supplier;
 import java.util.stream.Collectors;
+import org.apache.curator.framework.api.ACLProvider;
 import org.apache.solr.client.solrj.SolrClient;
 import org.apache.solr.client.solrj.cloud.SolrCloudManager;
 import org.apache.solr.client.solrj.impl.CloudHttp2SolrClient;
@@ -68,7 +69,6 @@
 import org.apache.solr.common.SolrException;
 import org.apache.solr.common.SolrException.ErrorCode;
 import org.apache.solr.common.cloud.ClusterState;
-import org.apache.solr.common.cloud.DefaultConnectionStrategy;
 import org.apache.solr.common.cloud.DefaultZkACLProvider;
 import org.apache.solr.common.cloud.DefaultZkCredentialsInjector;
 import org.apache.solr.common.cloud.DefaultZkCredentialsProvider;
@@ -84,7 +84,6 @@
 import org.apache.solr.common.cloud.Slice;
 import org.apache.solr.common.cloud.SolrZkClient;
 import org.apache.solr.common.cloud.ZkACLProvider;
-import org.apache.solr.common.cloud.ZkClientConnectionStrategy;
 import org.apache.solr.common.cloud.ZkCoreNodeProps;
 import org.apache.solr.common.cloud.ZkCredentialsInjector;
 import org.apache.solr.common.cloud.ZkCredentialsProvider;
@@ -122,7 +121,6 @@
 import org.apache.zookeeper.KeeperException;
 import org.apache.zookeeper.KeeperException.NoNodeException;
 import org.apache.zookeeper.KeeperException.SessionExpiredException;
-import org.apache.zookeeper.Op;
 import org.apache.zookeeper.WatchedEvent;
 import org.apache.zookeeper.Watcher;
 import org.apache.zookeeper.data.ACL;
@@ -233,6 +231,8 @@ public String toString() {
 
   private boolean genericCoreNodeNames;
 
+  private int clientTimeout;
+
   private volatile boolean isClosed;
 
   private final ConcurrentHashMap<String, Throwable> replicasMetTragicEvent =
@@ -329,12 +329,6 @@ public ZkController(
     this.leaderVoteWait = cloudConfig.getLeaderVoteWait();
     this.leaderConflictResolveWait = cloudConfig.getLeaderConflictResolveWait();
 
-    int clientTimeout = cloudConfig.getZkClientTimeout();
-
-    String connectionStrategy = System.getProperty("solr.zookeeper.connectionStrategy");
-    ZkClientConnectionStrategy strat =
-        ZkClientConnectionStrategy.forName(connectionStrategy, new DefaultConnectionStrategy());
-
     String zkCredentialsInjectorClass = cloudConfig.getZkCredentialsInjectorClass();
     ZkCredentialsInjector zkCredentialsInjector =
         StrUtils.isNullOrEmpty(zkCredentialsInjectorClass)
@@ -342,6 +336,8 @@ public ZkController(
             : cc.getResourceLoader()
                 .newInstance(zkCredentialsInjectorClass, ZkCredentialsInjector.class);
 
+    this.clientTimeout = cloudConfig.getZkClientTimeout();
+
     String zkACLProviderClass = cloudConfig.getZkACLProviderClass();
     ZkACLProvider zkACLProvider =
         StrUtils.isNullOrEmpty(zkACLProviderClass)
@@ -357,7 +353,6 @@ public ZkController(
                 .newInstance(zkCredentialsProviderClass, ZkCredentialsProvider.class);
 
     zkCredentialsProvider.setZkCredentialsInjector(zkCredentialsInjector);
-    strat.setZkCredentialsToAddAutomatically(zkCredentialsProvider);
     addOnReconnectListener(getConfigDirListener());
 
     String stateCompressionProviderClass = cloudConfig.getStateCompressorClass();
@@ -371,7 +366,6 @@ public ZkController(
             .withUrl(zkServerAddress)
             .withTimeout(clientTimeout, TimeUnit.MILLISECONDS)
             .withConnTimeOut(zkClientConnectTimeout, TimeUnit.MILLISECONDS)
-            .withConnStrategy(strat)
             .withReconnectListener(() -> onReconnect(descriptorsSupplier))
             .withBeforeConnect(() -> beforeReconnect(descriptorsSupplier))
             .withAclProvider(zkACLProvider)
@@ -419,8 +413,7 @@ private void beforeReconnect(Supplier<List<CoreDescriptor>> descriptorsSupplier)
     markAllAsNotLeader(descriptorsSupplier);
   }
 
-  private void onReconnect(Supplier<List<CoreDescriptor>> descriptorsSupplier)
-      throws SessionExpiredException {
+  private void onReconnect(Supplier<List<CoreDescriptor>> descriptorsSupplier) {
     // on reconnect, reload cloud info
     log.info("ZooKeeper session re-connected ... refreshing core states after session expiration.");
     clearZkCollectionTerms();
@@ -521,8 +514,6 @@ private void onReconnect(Supplier<List<CoreDescriptor>> descriptorsSupplier)
       // Restore the interrupted status
       Thread.currentThread().interrupt();
       throw new ZooKeeperException(ErrorCode.SERVER_ERROR, "", e);
-    } catch (SessionExpiredException e) {
-      throw e;
     } catch (Exception e) {
       log.error("Exception during reconnect", e);
       throw new ZooKeeperException(ErrorCode.SERVER_ERROR, "", e);
@@ -710,8 +701,10 @@ public void preClose() {
     this.isClosed = true;
 
     try {
-      this.removeEphemeralLiveNode();
-    } catch (AlreadyClosedException
+      if (getZkClient().isConnected()) {
+        this.removeEphemeralLiveNode();
+      }
+    } catch (IllegalStateException
         | SessionExpiredException
         | KeeperException.ConnectionLossException e) {
 
@@ -720,7 +713,7 @@ public void preClose() {
     }
 
     try {
-      if (getZkClient().getConnectionManager().isConnected()) {
+      if (getZkClient().isConnected()) {
         log.info("Publish this node as DOWN...");
         publishNodeAsDown(getNodeName());
       }
@@ -951,12 +944,12 @@ public static void createClusterZkNodes(SolrZkClient zkClient)
   private static void repairSecurityJson(SolrZkClient zkClient)
       throws KeeperException, InterruptedException {
     List<ACL> securityConfAcl = zkClient.getACL(ZkStateReader.SOLR_SECURITY_CONF_PATH, null, true);
-    ZkACLProvider aclProvider = zkClient.getZkACLProvider();
+    ACLProvider aclProvider = zkClient.getZkACLProvider();
 
     boolean tryUpdate = false;
 
     if (OPEN_ACL_UNSAFE.equals(securityConfAcl)) {
-      List<ACL> aclToAdd = aclProvider.getACLsToAdd(ZkStateReader.SOLR_SECURITY_CONF_PATH);
+      List<ACL> aclToAdd = aclProvider.getAclForPath(ZkStateReader.SOLR_SECURITY_CONF_PATH);
       if (OPEN_ACL_UNSAFE.equals(aclToAdd)) {
         log.warn(
             "Contents of zookeeper /security.json are world-readable;"
@@ -966,7 +959,7 @@ private static void repairSecurityJson(SolrZkClient zkClient)
       }
     } else if (aclProvider instanceof SecurityAwareZkACLProvider) {
       // Use Set to explicitly ignore order
-      Set<ACL> nonSecureACL = new HashSet<>(aclProvider.getACLsToAdd(null));
+      Set<ACL> nonSecureACL = new HashSet<>(aclProvider.getDefaultAcl());
       // case where security.json was not treated as a secure path
       if (nonSecureACL.equals(new HashSet<>(securityConfAcl))) {
         tryUpdate = true;
@@ -1070,7 +1063,7 @@ private void checkForExistingEphemeralNode() throws KeeperException, Interrupted
     }
 
     boolean deleted =
-        deletedLatch.await(zkClient.getZooKeeper().getSessionTimeout() * 2L, TimeUnit.MILLISECONDS);
+        deletedLatch.await(zkClient.getZkSessionTimeout() * 2L, TimeUnit.MILLISECONDS);
     if (!deleted) {
       throw new SolrException(
           ErrorCode.SERVER_ERROR,
@@ -1188,25 +1181,19 @@ private void createEphemeralLiveNode() throws KeeperException, InterruptedExcept
     String nodePath = ZkStateReader.LIVE_NODES_ZKNODE + "/" + nodeName;
     log.info("Register node as live in ZooKeeper:{}", nodePath);
     Map<NodeRoles.Role, String> roles = cc.nodeRoles.getRoles();
-    List<Op> ops = new ArrayList<>(roles.size() + 1);
-    ops.add(
-        Op.create(
-            nodePath,
-            null,
-            zkClient.getZkACLProvider().getACLsToAdd(nodePath),
-            CreateMode.EPHEMERAL));
+    List<SolrZkClient.CuratorOpBuilder> ops = new ArrayList<>(roles.size() + 1);
+    ops.add(op -> op.create().withMode(CreateMode.EPHEMERAL).forPath(nodePath));
 
     // Create the roles node as well
     roles.forEach(
         (role, mode) ->
             ops.add(
-                Op.create(
-                    NodeRoles.getZNodeForRoleMode(role, mode) + "/" + nodeName,
-                    null,
-                    zkClient.getZkACLProvider().getACLsToAdd(nodePath),
-                    CreateMode.EPHEMERAL)));
+                op ->
+                    op.create()
+                        .withMode(CreateMode.EPHEMERAL)
+                        .forPath(NodeRoles.getZNodeForRoleMode(role, mode) + "/" + nodeName)));
 
-    zkClient.multi(ops, true);
+    zkClient.multi(ops);
   }
 
   public void removeEphemeralLiveNode() throws KeeperException, InterruptedException {
@@ -1216,10 +1203,8 @@ public void removeEphemeralLiveNode() throws KeeperException, InterruptedExcepti
     String nodeName = getNodeName();
     String nodePath = ZkStateReader.LIVE_NODES_ZKNODE + "/" + nodeName;
     log.info("Remove node as live in ZooKeeper:{}", nodePath);
-    List<Op> ops = List.of(Op.delete(nodePath, -1));
-
     try {
-      zkClient.multi(ops, true);
+      zkClient.delete(nodePath, -1, true);
     } catch (NoNodeException e) {
 
     }
@@ -1511,7 +1496,7 @@ private String getLeader(final CloudDescriptor cloudDesc, int timeoutms) {
         leaderUrl = getLeaderProps(collection, cloudDesc.getShardId(), timeoutms).getCoreUrl();
       }
 
-    } catch (AlreadyClosedException e) {
+    } catch (IllegalStateException e) {
       throw e;
     } catch (Exception e) {
       log.error("Error getting leader from zk", e);
@@ -2539,7 +2524,7 @@ public static int persistConfigResourceToZooKeeper(
       } catch (NoNodeException e) {
         if (createIfNotExists) {
           try {
-            zkClient.create(resourceLocation, content, CreateMode.PERSISTENT, true);
+            zkClient.makePath(resourceLocation, content, CreateMode.PERSISTENT, true);
             latestVersion = 0; // just created so version must be zero
             touchConfDir(zkLoader);
           } catch (KeeperException.NodeExistsException nee) {
@@ -2892,7 +2877,7 @@ public Collection<String> publishNodeAsDown(String nodeName) {
                 m ->
                     m.put(Overseer.QUEUE_OPERATION, OverseerAction.DOWNNODE.toLower())
                         .put(ZkStateReader.NODE_NAME_PROP, nodeName));
-      } catch (AlreadyClosedException e) {
+      } catch (IllegalStateException e) {
         log.info(
             "Not publishing node as DOWN because a resource required to do so is already closed.");
       } catch (InterruptedException e) {
diff --git a/solr/core/src/java/org/apache/solr/cloud/ZkDistributedQueue.java b/solr/core/src/java/org/apache/solr/cloud/ZkDistributedQueue.java
index 57db14740c5..f3e06f90cb8 100644
--- a/solr/core/src/java/org/apache/solr/cloud/ZkDistributedQueue.java
+++ b/solr/core/src/java/org/apache/solr/cloud/ZkDistributedQueue.java
@@ -32,16 +32,15 @@
 import java.util.concurrent.locks.Condition;
 import java.util.concurrent.locks.ReentrantLock;
 import java.util.function.Predicate;
+import org.apache.curator.framework.api.transaction.CuratorTransactionResult;
 import org.apache.solr.client.solrj.cloud.DistributedQueue;
 import org.apache.solr.common.SolrException;
 import org.apache.solr.common.SolrException.ErrorCode;
-import org.apache.solr.common.cloud.ConnectionManager.IsClosed;
 import org.apache.solr.common.cloud.SolrZkClient;
 import org.apache.solr.common.cloud.ZkMaintenanceUtils;
 import org.apache.solr.common.util.Pair;
 import org.apache.zookeeper.CreateMode;
 import org.apache.zookeeper.KeeperException;
-import org.apache.zookeeper.Op;
 import org.apache.zookeeper.WatchedEvent;
 import org.apache.zookeeper.Watcher;
 import org.apache.zookeeper.data.Stat;
@@ -123,15 +122,6 @@ public ZkDistributedQueue(SolrZkClient zookeeper, String dir, Stats stats) {
   }
 
   public ZkDistributedQueue(SolrZkClient zookeeper, String dir, Stats stats, int maxQueueSize) {
-    this(zookeeper, dir, stats, maxQueueSize, null);
-  }
-
-  public ZkDistributedQueue(
-      SolrZkClient zookeeper,
-      String dir,
-      Stats stats,
-      int maxQueueSize,
-      IsClosed higherLevelIsClosed) {
     this.dir = dir;
 
     try {
@@ -246,24 +236,19 @@ public byte[] remove() throws NoSuchElementException, KeeperException, Interrupt
 
   public void remove(Collection<String> paths) throws KeeperException, InterruptedException {
     if (paths.isEmpty()) return;
-    List<Op> ops = new ArrayList<>();
+    List<SolrZkClient.CuratorOpBuilder> ops = new ArrayList<>();
     for (String path : paths) {
-      ops.add(Op.delete(dir + "/" + path, -1));
+      ops.add(op -> op.delete().withVersion(-1).forPath(dir + "/" + path));
     }
     for (int from = 0; from < ops.size(); from += 1000) {
       int to = Math.min(from + 1000, ops.size());
       if (from < to) {
-        try {
-          zookeeper.multi(ops.subList(from, to), true);
-        } catch (KeeperException.NoNodeException e) {
-          // don't know which nodes are not exist, so try to delete one by one node
-          for (int j = from; j < to; j++) {
+        Collection<CuratorTransactionResult> results = zookeeper.multi(ops.subList(from, to));
+        for (CuratorTransactionResult result : results) {
+          if (result.getError() != 0) {
             try {
-              zookeeper.delete(ops.get(j).getPath(), -1, true);
-            } catch (KeeperException.NoNodeException e2) {
-              if (log.isDebugEnabled()) {
-                log.debug("Can not remove node which is not exist : {}", ops.get(j).getPath());
-              }
+              zookeeper.delete(result.getForPath(), -1, true);
+            } catch (KeeperException.NoNodeException ignored) {
             }
           }
         }
diff --git a/solr/core/src/java/org/apache/solr/cloud/ZkShardTerms.java b/solr/core/src/java/org/apache/solr/cloud/ZkShardTerms.java
index b15567bf082..83f8885325b 100644
--- a/solr/core/src/java/org/apache/solr/cloud/ZkShardTerms.java
+++ b/solr/core/src/java/org/apache/solr/cloud/ZkShardTerms.java
@@ -22,7 +22,6 @@
 import java.util.HashSet;
 import java.util.Map;
 import java.util.Set;
-import java.util.concurrent.TimeoutException;
 import java.util.concurrent.atomic.AtomicBoolean;
 import java.util.concurrent.atomic.AtomicReference;
 import java.util.function.Function;
@@ -398,16 +397,6 @@ private void retryRegisterWatcher() {
         return;
       } catch (KeeperException e) {
         log.warn("Failed watching shard term for collection: {}, retrying!", collection, e);
-        try {
-          zkClient.getConnectionManager().waitForConnected(zkClient.getZkClientTimeout());
-        } catch (TimeoutException te) {
-          if (Thread.interrupted()) {
-            throw new SolrException(
-                SolrException.ErrorCode.SERVER_ERROR,
-                "Error watching shard term for collection: " + collection,
-                te);
-          }
-        }
       }
     }
   }
@@ -416,6 +405,10 @@ private void retryRegisterWatcher() {
   private void registerWatcher() throws KeeperException {
     Watcher watcher =
         event -> {
+          // Don't do anything if we are closed
+          if (isClosed.get()) {
+            return;
+          }
           // session events are not change events, and do not remove the watcher
           if (Watcher.Event.EventType.None == event.getType()) {
             return;
diff --git a/solr/core/src/java/org/apache/solr/core/ZkContainer.java b/solr/core/src/java/org/apache/solr/core/ZkContainer.java
index 02e6d4e628b..6dd7b2d571d 100644
--- a/solr/core/src/java/org/apache/solr/core/ZkContainer.java
+++ b/solr/core/src/java/org/apache/solr/core/ZkContainer.java
@@ -32,7 +32,6 @@
 import org.apache.solr.client.solrj.impl.SolrZkClientTimeout;
 import org.apache.solr.cloud.SolrZkServer;
 import org.apache.solr.cloud.ZkController;
-import org.apache.solr.common.AlreadyClosedException;
 import org.apache.solr.common.SolrException;
 import org.apache.solr.common.cloud.ClusterProperties;
 import org.apache.solr.common.cloud.Replica;
@@ -217,7 +216,7 @@ public void registerInZk(final SolrCore core, boolean background, boolean skipRe
               log.error("Interrupted", e);
             } catch (KeeperException e) {
               log.error("KeeperException registering core {}", core.getName(), e);
-            } catch (AlreadyClosedException ignore) {
+            } catch (IllegalStateException ignore) {
 
             } catch (Exception e) {
               log.error("Exception registering core {}", core.getName(), e);
diff --git a/solr/core/src/java/org/apache/solr/handler/component/SearchHandler.java b/solr/core/src/java/org/apache/solr/handler/component/SearchHandler.java
index 28db4f4478c..2c1503a907f 100644
--- a/solr/core/src/java/org/apache/solr/handler/component/SearchHandler.java
+++ b/solr/core/src/java/org/apache/solr/handler/component/SearchHandler.java
@@ -323,9 +323,7 @@ public ShardHandler getAndPrepShardHandler(SolrQueryRequest req, ResponseBuilder
       boolean requireZkConnected =
           shardsTolerant != null && shardsTolerant.equals(ShardParams.REQUIRE_ZK_CONNECTED);
       ZkController zkController = cc.getZkController();
-      boolean zkConnected =
-          zkController != null
-              && !zkController.getZkClient().getConnectionManager().isLikelyExpired();
+      boolean zkConnected = zkController != null && zkController.getZkClient().isConnected();
       if (requireZkConnected && false == zkConnected) {
         throw new SolrException(SolrException.ErrorCode.SERVER_ERROR, "ZooKeeper is not connected");
       } else {
diff --git a/solr/core/src/java/org/apache/solr/update/processor/DistributedZkUpdateProcessor.java b/solr/core/src/java/org/apache/solr/update/processor/DistributedZkUpdateProcessor.java
index 35d96a7c14a..8c8c3ce6a08 100644
--- a/solr/core/src/java/org/apache/solr/update/processor/DistributedZkUpdateProcessor.java
+++ b/solr/core/src/java/org/apache/solr/update/processor/DistributedZkUpdateProcessor.java
@@ -1394,7 +1394,7 @@ private void zkCheck(UpdateCommand updateCommand) {
       return;
     }
 
-    if (!zkController.getZkClient().getConnectionManager().isLikelyExpired()) {
+    if (zkController.isConnected()) {
       return;
     }
 
diff --git a/solr/core/src/test/org/apache/solr/cli/ZkSubcommandsTest.java b/solr/core/src/test/org/apache/solr/cli/ZkSubcommandsTest.java
index 5cf76769d99..302741b72d1 100644
--- a/solr/core/src/test/org/apache/solr/cli/ZkSubcommandsTest.java
+++ b/solr/core/src/test/org/apache/solr/cli/ZkSubcommandsTest.java
@@ -178,7 +178,7 @@ public void testPutCompressed() throws Exception {
     ZkCpTool tool = new ZkCpTool();
     assertEquals(0, runTool(args, tool));
 
-    assertArrayEquals(zkClient.getZooKeeper().getData("/state.json", null, null), expected);
+    assertArrayEquals(zkClient.getCuratorFramework().getData().forPath("/state.json"), expected);
 
     // test re-put to existing
     data = "my data deux";
@@ -201,13 +201,13 @@ public void testPutCompressed() throws Exception {
         };
     assertEquals(0, runTool(args, tool));
 
-    byte[] fromZk = zkClient.getZooKeeper().getData("/state.json", null, null);
+    byte[] fromZk = zkClient.getCuratorFramework().getData().forPath("/state.json");
     byte[] fromLoc =
         new ZLibCompressor()
             .compressBytes(Files.readAllBytes(Path.of(localFile.getAbsolutePath())));
     assertArrayEquals("Should get back what we put in ZK", fromLoc, fromZk);
 
-    assertArrayEquals(zkClient.getZooKeeper().getData("/state.json", null, null), expected);
+    assertArrayEquals(zkClient.getCuratorFramework().getData().forPath("/state.json"), expected);
   }
 
   @Test
@@ -272,7 +272,7 @@ public void testPutFileCompressed() throws Exception {
     ZkCpTool tool = new ZkCpTool();
     assertEquals(0, runTool(args, tool));
 
-    byte[] fromZk = zkClient.getZooKeeper().getData("/state.json", null, null);
+    byte[] fromZk = zkClient.getCuratorFramework().getData().forPath("/state.json");
     Path locFile = Path.of(SOLR_HOME, "solr-stress-new.xml");
     byte[] fromLoc = new ZLibCompressor().compressBytes(Files.readAllBytes(locFile));
     assertArrayEquals("Should get back what we put in ZK", fromLoc, fromZk);
@@ -280,7 +280,7 @@ public void testPutFileCompressed() throws Exception {
     // Lets do it again
     assertEquals(0, runTool(args, tool));
 
-    fromZk = zkClient.getZooKeeper().getData("/state.json", null, null);
+    fromZk = zkClient.getCuratorFramework().getData().forPath("/state.json");
     locFile = Path.of(SOLR_HOME, "solr-stress-new.xml");
     fromLoc = new ZLibCompressor().compressBytes(Files.readAllBytes(locFile));
     assertArrayEquals("Should get back what we put in ZK", fromLoc, fromZk);
diff --git a/solr/core/src/test/org/apache/solr/cloud/ConnectionManagerTest.java b/solr/core/src/test/org/apache/solr/cloud/ConnectionManagerTest.java
deleted file mode 100644
index d48ba5bc3cc..00000000000
--- a/solr/core/src/test/org/apache/solr/cloud/ConnectionManagerTest.java
+++ /dev/null
@@ -1,186 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements.  See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License.  You may obtain a copy of the License at
- *
- *     http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.solr.cloud;
-
-import java.io.IOException;
-import java.nio.file.Path;
-import java.util.concurrent.Executors;
-import java.util.concurrent.ScheduledExecutorService;
-import java.util.concurrent.TimeUnit;
-import java.util.concurrent.TimeoutException;
-import org.apache.solr.SolrTestCaseJ4;
-import org.apache.solr.common.cloud.ConnectionManager;
-import org.apache.solr.common.cloud.SolrZkClient;
-import org.apache.solr.common.util.SolrNamedThreadFactory;
-import org.apache.zookeeper.TestableZooKeeper;
-import org.apache.zookeeper.WatchedEvent;
-import org.apache.zookeeper.Watcher;
-import org.apache.zookeeper.Watcher.Event.EventType;
-import org.apache.zookeeper.Watcher.Event.KeeperState;
-import org.apache.zookeeper.ZooKeeper;
-import org.junit.Ignore;
-import org.junit.Test;
-
-public class ConnectionManagerTest extends SolrTestCaseJ4 {
-
-  static final int TIMEOUT = 3000;
-
-  @Ignore
-  public void testConnectionManager() throws Exception {
-
-    // setup a SolrZkClient to do some getBaseUrlForNodeName testing
-    Path zkDir = createTempDir("zkData");
-    ZkTestServer server = new ZkTestServer(zkDir);
-    try {
-      server.run();
-
-      SolrZkClient zkClient =
-          new SolrZkClient.Builder()
-              .withUrl(server.getZkAddress())
-              .withTimeout(TIMEOUT, TimeUnit.MILLISECONDS)
-              .build();
-      ConnectionManager cm = zkClient.getConnectionManager();
-      try {
-        assertFalse(cm.isLikelyExpired());
-
-        ZooKeeper zk = zkClient.getZooKeeper();
-        assertTrue(zk instanceof TestableZooKeeper);
-        ((TestableZooKeeper) zk).testableConnloss();
-        server.expire(zkClient.getZooKeeper().getSessionId());
-
-        Thread.sleep(TIMEOUT);
-
-        assertTrue(cm.isLikelyExpired());
-      } finally {
-        cm.close();
-        zkClient.close();
-      }
-    } finally {
-      server.shutdown();
-    }
-  }
-
-  public void testLikelyExpired() throws Exception {
-
-    // setup a SolrZkClient to do some getBaseUrlForNodeName testing
-    Path zkDir = createTempDir("zkData");
-    ZkTestServer server = new ZkTestServer(zkDir);
-    try {
-      server.run();
-
-      SolrZkClient zkClient =
-          new SolrZkClient.Builder()
-              .withUrl(server.getZkAddress())
-              .withTimeout(TIMEOUT, TimeUnit.MILLISECONDS)
-              .build();
-      ConnectionManager cm = zkClient.getConnectionManager();
-      try {
-        assertFalse(cm.isLikelyExpired());
-        assertTrue(cm.isConnectedAndNotClosed());
-        cm.process(new WatchedEvent(EventType.None, KeeperState.Disconnected, ""));
-        // disconnect shouldn't immediately set likelyExpired
-        assertFalse(cm.isConnectedAndNotClosed());
-        assertFalse(cm.isLikelyExpired());
-
-        // but it should after the timeout
-        Thread.sleep((long) (zkClient.getZkClientTimeout() * 1.5));
-        assertFalse(cm.isConnectedAndNotClosed());
-        assertTrue(cm.isLikelyExpired());
-
-        // even if we disconnect immediately again
-        cm.process(new WatchedEvent(EventType.None, KeeperState.Disconnected, ""));
-        assertFalse(cm.isConnectedAndNotClosed());
-        assertTrue(cm.isLikelyExpired());
-
-        // reconnect -- should no longer be likely expired
-        cm.process(new WatchedEvent(EventType.None, KeeperState.SyncConnected, ""));
-        assertFalse(cm.isLikelyExpired());
-        assertTrue(cm.isConnectedAndNotClosed());
-      } finally {
-        cm.close();
-        zkClient.close();
-      }
-    } finally {
-      server.shutdown();
-    }
-  }
-
-  @Test
-  public void testReconnectWhenZkDisappeared() throws Exception {
-    ScheduledExecutorService executor =
-        Executors.newSingleThreadScheduledExecutor(
-            new SolrNamedThreadFactory("connectionManagerTest"));
-
-    // setup a SolrZkClient to do some getBaseUrlForNodeName testing
-    Path zkDir = createTempDir("zkData");
-    ZkTestServer server = new ZkTestServer(zkDir);
-    try {
-      server.run();
-
-      MockZkClientConnectionStrategy strategy = new MockZkClientConnectionStrategy();
-      SolrZkClient zkClient =
-          new SolrZkClient.Builder()
-              .withUrl(server.getZkAddress())
-              .withTimeout(TIMEOUT, TimeUnit.MILLISECONDS)
-              .withConnStrategy(strategy)
-              .build();
-      ConnectionManager cm = zkClient.getConnectionManager();
-
-      try {
-        assertFalse(cm.isLikelyExpired());
-        assertTrue(cm.isConnectedAndNotClosed());
-
-        // reconnect -- should no longer be likely expired
-        cm.process(new WatchedEvent(EventType.None, KeeperState.Expired, ""));
-        assertFalse(cm.isLikelyExpired());
-        assertTrue(cm.isConnectedAndNotClosed());
-        assertTrue(strategy.isExceptionThrow());
-      } finally {
-        cm.close();
-        zkClient.close();
-        executor.shutdown();
-      }
-    } finally {
-      server.shutdown();
-    }
-  }
-
-  private static class MockZkClientConnectionStrategy extends TestConnectionStrategy {
-    int called = 0;
-    boolean exceptionThrown = false;
-
-    @Override
-    public void reconnect(
-        final String serverAddress,
-        final int zkClientTimeout,
-        final Watcher watcher,
-        final ZkUpdate updater)
-        throws IOException, InterruptedException, TimeoutException {
-
-      if (called++ < 1) {
-        exceptionThrown = true;
-        throw new IOException("Testing");
-      }
-
-      super.reconnect(serverAddress, zkClientTimeout, watcher, updater);
-    }
-
-    public boolean isExceptionThrow() {
-      return exceptionThrown;
-    }
-  }
-}
diff --git a/solr/core/src/test/org/apache/solr/cloud/DistributedQueueTest.java b/solr/core/src/test/org/apache/solr/cloud/DistributedQueueTest.java
index 9a57f4d6c9d..213a5db6d6d 100644
--- a/solr/core/src/test/org/apache/solr/cloud/DistributedQueueTest.java
+++ b/solr/core/src/test/org/apache/solr/cloud/DistributedQueueTest.java
@@ -19,6 +19,7 @@
 import java.nio.charset.Charset;
 import java.nio.charset.StandardCharsets;
 import java.util.NoSuchElementException;
+import java.util.concurrent.CountDownLatch;
 import java.util.concurrent.ExecutorService;
 import java.util.concurrent.Future;
 import java.util.concurrent.TimeUnit;
@@ -26,6 +27,7 @@
 import java.util.function.Predicate;
 import org.apache.solr.SolrTestCaseJ4;
 import org.apache.solr.client.solrj.cloud.DistributedQueue;
+import org.apache.solr.common.cloud.OnDisconnect;
 import org.apache.solr.common.cloud.SolrZkClient;
 import org.apache.solr.common.util.ExecutorUtil;
 import org.apache.solr.common.util.SolrNamedThreadFactory;
@@ -289,18 +291,19 @@ public void testPeekElements() throws Exception {
   }
 
   private void forceSessionExpire() throws InterruptedException, TimeoutException {
-    long sessionId = zkClient.getZooKeeper().getSessionId();
+    final CountDownLatch hasDisconnected = new CountDownLatch(1);
+    zkClient
+        .getCuratorFramework()
+        .getConnectionStateListenable()
+        .addListener((OnDisconnect) hasDisconnected::countDown);
+    long sessionId = zkClient.getZkSessionId();
     zkServer.expire(sessionId);
-    zkClient.getConnectionManager().waitForDisconnected(10000);
-    zkClient.getConnectionManager().waitForConnected(10000);
-    for (int i = 0; i < 100; ++i) {
-      if (zkClient.isConnected()) {
-        break;
-      }
-      Thread.sleep(50);
-    }
+    hasDisconnected.await(10, TimeUnit.SECONDS);
+    assertEquals(
+        "ZK Client did not disconnect after session expiration", 0, hasDisconnected.getCount());
+    zkClient.getCuratorFramework().blockUntilConnected(10, TimeUnit.SECONDS);
     assertTrue(zkClient.isConnected());
-    assertNotEquals(sessionId, zkClient.getZooKeeper().getSessionId());
+    assertNotEquals(sessionId, zkClient.getZkSessionId());
   }
 
   protected ZkDistributedQueue makeDistributedQueue(String dqZNode) throws Exception {
diff --git a/solr/core/src/test/org/apache/solr/cloud/LeaderElectionTest.java b/solr/core/src/test/org/apache/solr/cloud/LeaderElectionTest.java
index 44e226afdf0..d4a61abcc60 100644
--- a/solr/core/src/test/org/apache/solr/cloud/LeaderElectionTest.java
+++ b/solr/core/src/test/org/apache/solr/cloud/LeaderElectionTest.java
@@ -28,6 +28,7 @@
 import java.util.concurrent.Executors;
 import java.util.concurrent.ScheduledExecutorService;
 import java.util.concurrent.TimeUnit;
+import org.apache.curator.test.KillSession;
 import org.apache.solr.SolrTestCaseJ4;
 import org.apache.solr.common.cloud.OnReconnect;
 import org.apache.solr.common.cloud.SolrZkClient;
@@ -39,8 +40,6 @@
 import org.apache.zookeeper.KeeperException;
 import org.apache.zookeeper.KeeperException.NoNodeException;
 import org.apache.zookeeper.KeeperException.SessionExpiredException;
-import org.apache.zookeeper.TestableZooKeeper;
-import org.apache.zookeeper.ZooKeeper;
 import org.junit.Test;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
@@ -119,7 +118,7 @@ public ElectorSetup(OnReconnect onReconnect) {
               .build();
       zkStateReader = new ZkStateReader(zkClient);
       elector = new LeaderElector(zkClient);
-      zkController = MockSolrSource.makeSimpleMock(null, zkStateReader, null);
+      zkController = MockSolrSource.makeSimpleMock(null, zkStateReader, zkClient);
     }
 
     public void close() {
@@ -187,9 +186,10 @@ private void setupOnConnect() throws InterruptedException, KeeperException {
     public void run() {
       try {
         setupOnConnect();
+      } catch (IllegalStateException ignored) {
+        return;
       } catch (Throwable e) {
         log.error("setup failed", e);
-        es.close();
         return;
       }
 
@@ -478,8 +478,7 @@ public void testStressElection() throws Exception {
     final List<ClientThread> threads = Collections.synchronizedList(new ArrayList<ClientThread>());
 
     // start with a leader
-    ClientThread thread1 = null;
-    thread1 = new ClientThread("shard1", 0);
+    ClientThread thread1 = new ClientThread("shard1", 0);
     threads.add(thread1);
     scheduler.schedule(thread1, 0, TimeUnit.MILLISECONDS);
 
@@ -539,11 +538,15 @@ public void run() {
                   int j;
                   j = random().nextInt(threads.size());
                   try {
-                    ZooKeeper zk = threads.get(j).es.zkClient.getZooKeeper();
-                    assertTrue(zk instanceof TestableZooKeeper);
-                    ((TestableZooKeeper) zk).testableConnloss();
                     if (random().nextBoolean()) {
-                      server.expire(zk.getSessionId());
+                      KillSession.kill(
+                          threads
+                              .get(j)
+                              .es
+                              .zkClient
+                              .getCuratorFramework()
+                              .getZookeeperClient()
+                              .getZooKeeper());
                     }
                   } catch (Exception e) {
                     log.error("error expiring session", e);
@@ -555,36 +558,43 @@ public void run() {
               }
             });
 
-    scheduleThread.start();
-    connLossThread.start();
-    killThread.start();
+    try {
+      scheduleThread.start();
+      connLossThread.start();
+      killThread.start();
 
-    Thread.sleep(4000);
+      Thread.sleep(4000);
 
-    stopStress = true;
+      stopStress = true;
 
-    scheduleThread.interrupt();
-    connLossThread.interrupt();
-    killThread.interrupt();
+      scheduleThread.interrupt();
+      connLossThread.interrupt();
+      killThread.interrupt();
 
-    scheduleThread.join();
-    scheduler.shutdownNow();
+      scheduleThread.join();
+      scheduler.shutdownNow();
 
-    connLossThread.join();
-    killThread.join();
+      connLossThread.join();
+      killThread.join();
 
-    int seq = threads.get(getLeaderThread()).getSeq();
+      int seq = threads.get(getLeaderThread()).getSeq();
 
-    // we have a leader we know, TODO: lets check some other things
+      // we have a leader we know, TODO: lets check some other things
+    } finally {
+      // cleanup any threads still running
+      for (ClientThread thread : threads) {
+        thread.close();
+      }
 
-    // cleanup any threads still running
-    for (ClientThread thread : threads) {
-      thread.es.zkClient.getZooKeeper().close();
-      thread.close();
-    }
+      // cleanup any threads still running
+      for (ClientThread thread : threads) {
+        thread.es.zkClient.close();
+        thread.close();
+      }
 
-    for (Thread thread : threads) {
-      thread.join();
+      for (Thread thread : threads) {
+        thread.join();
+      }
     }
   }
 
diff --git a/solr/core/src/test/org/apache/solr/cloud/MultiThreadedOCPTest.java b/solr/core/src/test/org/apache/solr/cloud/MultiThreadedOCPTest.java
index e67fa4f280e..833f28a1a63 100644
--- a/solr/core/src/test/org/apache/solr/cloud/MultiThreadedOCPTest.java
+++ b/solr/core/src/test/org/apache/solr/cloud/MultiThreadedOCPTest.java
@@ -246,8 +246,7 @@ public void testTaskExclusivity() throws Exception, SolrServerException {
   @Test
   public void testDeduplicationOfSubmittedTasks() throws IOException, SolrServerException {
     try (SolrClient client = createNewSolrClient("", getBaseUrl(jettys.get(0)))) {
-      CollectionAdminRequest.createCollection("ocptest_shardsplit2", "conf1", 3, 1)
-          .processAsync("3000", client);
+      CollectionAdminRequest.createCollection("ocptest_shardsplit2", "conf1", 3, 1).process(client);
 
       SplitShard splitShardRequest =
           CollectionAdminRequest.splitShard("ocptest_shardsplit2").setShardName(SHARD1);
diff --git a/solr/core/src/test/org/apache/solr/cloud/OutOfBoxZkACLAndCredentialsProvidersTest.java b/solr/core/src/test/org/apache/solr/cloud/OutOfBoxZkACLAndCredentialsProvidersTest.java
index 8c45324946c..b0188069562 100644
--- a/solr/core/src/test/org/apache/solr/cloud/OutOfBoxZkACLAndCredentialsProvidersTest.java
+++ b/solr/core/src/test/org/apache/solr/cloud/OutOfBoxZkACLAndCredentialsProvidersTest.java
@@ -152,7 +152,7 @@ public void testOpenACLUnsafeAllover() throws Exception {
 
   protected void assertOpenACLUnsafeAllover(
       SolrZkClient zkClient, String path, List<String> verifiedList) throws Exception {
-    List<ACL> acls = zkClient.getZooKeeper().getACL(path, new Stat());
+    List<ACL> acls = zkClient.getACL(path, new Stat(), true);
     if (log.isInfoEnabled()) {
       log.info("Verifying {}", path);
     }
diff --git a/solr/core/src/test/org/apache/solr/cloud/OverriddenZkACLAndCredentialsProvidersTest.java b/solr/core/src/test/org/apache/solr/cloud/OverriddenZkACLAndCredentialsProvidersTest.java
index 79fdac41acf..5810c50fe17 100644
--- a/solr/core/src/test/org/apache/solr/cloud/OverriddenZkACLAndCredentialsProvidersTest.java
+++ b/solr/core/src/test/org/apache/solr/cloud/OverriddenZkACLAndCredentialsProvidersTest.java
@@ -29,15 +29,17 @@
 import java.util.ArrayList;
 import java.util.List;
 import java.util.concurrent.TimeUnit;
+import org.apache.curator.framework.AuthInfo;
+import org.apache.curator.framework.api.ACLProvider;
 import org.apache.solr.SolrTestCaseJ4;
 import org.apache.solr.common.cloud.DigestZkACLProvider;
 import org.apache.solr.common.cloud.DigestZkCredentialsProvider;
 import org.apache.solr.common.cloud.SecurityAwareZkACLProvider;
 import org.apache.solr.common.cloud.SolrZkClient;
 import org.apache.solr.common.cloud.VMParamsZkCredentialsInjector;
-import org.apache.solr.common.cloud.ZkACLProvider;
 import org.apache.solr.common.cloud.ZkCredentialsInjector;
 import org.apache.solr.common.cloud.ZkCredentialsProvider;
+import org.apache.solr.common.util.StrUtils;
 import org.apache.zookeeper.CreateMode;
 import org.junit.AfterClass;
 import org.junit.BeforeClass;
@@ -80,50 +82,57 @@ public void setUp() throws Exception {
 
     System.setProperty("zkHost", zkServer.getZkAddress());
 
-    SolrZkClient zkClient =
+    try (SolrZkClient zkClient =
         new SolrZkClientFactoryUsingCompletelyNewProviders(
                 "connectAndAllACLUsername",
                 "connectAndAllACLPassword",
                 "readonlyACLUsername",
                 "readonlyACLPassword")
-            .getSolrZkClient(zkServer.getZkHost(), AbstractZkTestCase.TIMEOUT);
-    zkClient.makePath("/solr", false, true);
-    zkClient.close();
+            .getSolrZkClient(zkServer.getZkHost(), AbstractZkTestCase.TIMEOUT)) {
+      zkClient.makePath("/solr", false, true);
+    }
 
-    zkClient =
+    try (SolrZkClient zkClient =
         new SolrZkClientFactoryUsingCompletelyNewProviders(
                 "connectAndAllACLUsername",
                 "connectAndAllACLPassword",
                 "readonlyACLUsername",
                 "readonlyACLPassword")
-            .getSolrZkClient(zkServer.getZkAddress(), AbstractZkTestCase.TIMEOUT);
-    zkClient.create(
-        "/protectedCreateNode", "content".getBytes(DATA_ENCODING), CreateMode.PERSISTENT, false);
-    zkClient.makePath(
-        "/protectedMakePathNode", "content".getBytes(DATA_ENCODING), CreateMode.PERSISTENT, false);
-    zkClient.create(
-        SecurityAwareZkACLProvider.SECURITY_ZNODE_PATH,
-        "content".getBytes(DATA_ENCODING),
-        CreateMode.PERSISTENT,
-        false);
-    zkClient.close();
-
-    zkClient =
+            .getSolrZkClient(zkServer.getZkAddress(), AbstractZkTestCase.TIMEOUT)) {
+      zkClient.create(
+          "/protectedCreateNode", "content".getBytes(DATA_ENCODING), CreateMode.PERSISTENT, false);
+      zkClient.makePath(
+          "/protectedMakePathNode",
+          "content".getBytes(DATA_ENCODING),
+          CreateMode.PERSISTENT,
+          false);
+      zkClient.create(
+          SecurityAwareZkACLProvider.SECURITY_ZNODE_PATH,
+          "content".getBytes(DATA_ENCODING),
+          CreateMode.PERSISTENT,
+          false);
+    }
+
+    try (SolrZkClient zkClient =
         new SolrZkClientFactoryUsingCompletelyNewProviders(null, null, null, null)
-            .getSolrZkClient(zkServer.getZkAddress(), AbstractZkTestCase.TIMEOUT);
-    zkClient
-        .getZooKeeper()
-        .addAuthInfo(
-            "digest",
-            ("connectAndAllACLUsername:connectAndAllACLPassword").getBytes(DATA_ENCODING));
-    zkClient.create(
-        "/unprotectedCreateNode", "content".getBytes(DATA_ENCODING), CreateMode.PERSISTENT, false);
-    zkClient.makePath(
-        "/unprotectedMakePathNode",
-        "content".getBytes(DATA_ENCODING),
-        CreateMode.PERSISTENT,
-        false);
-    zkClient.close();
+            .getSolrZkClient(
+                zkServer.getZkAddress(),
+                AbstractZkTestCase.TIMEOUT,
+                new AuthInfo(
+                    "digest",
+                    ("connectAndAllACLUsername:connectAndAllACLPassword")
+                        .getBytes(DATA_ENCODING)))) {
+      zkClient.create(
+          "/unprotectedCreateNode",
+          "content".getBytes(DATA_ENCODING),
+          CreateMode.PERSISTENT,
+          false);
+      zkClient.makePath(
+          "/unprotectedMakePathNode",
+          "content".getBytes(DATA_ENCODING),
+          CreateMode.PERSISTENT,
+          false);
+    }
 
     if (log.isInfoEnabled()) {
       log.info("####SETUP_END {}", getTestName());
@@ -141,58 +150,46 @@ public void tearDown() throws Exception {
 
   @Test
   public void testNoCredentialsSolrZkClientFactoryUsingCompletelyNewProviders() throws Exception {
-    SolrZkClient zkClient =
+    try (SolrZkClient zkClient =
         new SolrZkClientFactoryUsingCompletelyNewProviders(null, null, null, null)
-            .getSolrZkClient(zkServer.getZkAddress(), AbstractZkTestCase.TIMEOUT);
-    try {
+            .getSolrZkClient(zkServer.getZkAddress(), AbstractZkTestCase.TIMEOUT)) {
       AbstractDigestZkACLAndCredentialsProvidersTestBase.doTest(
           zkClient, false, false, false, false, false, false, false, false, false, false);
-    } finally {
-      zkClient.close();
     }
   }
 
   @Test
   public void testWrongCredentialsSolrZkClientFactoryUsingCompletelyNewProviders()
       throws Exception {
-    SolrZkClient zkClient =
+    try (SolrZkClient zkClient =
         new SolrZkClientFactoryUsingCompletelyNewProviders(
                 "connectAndAllACLUsername", "connectAndAllACLPasswordWrong", null, null)
-            .getSolrZkClient(zkServer.getZkAddress(), AbstractZkTestCase.TIMEOUT);
-    try {
+            .getSolrZkClient(zkServer.getZkAddress(), AbstractZkTestCase.TIMEOUT)) {
       AbstractDigestZkACLAndCredentialsProvidersTestBase.doTest(
           zkClient, false, false, false, false, false, false, false, false, false, false);
-    } finally {
-      zkClient.close();
     }
   }
 
   @Test
   public void testAllCredentialsSolrZkClientFactoryUsingCompletelyNewProviders() throws Exception {
-    SolrZkClient zkClient =
+    try (SolrZkClient zkClient =
         new SolrZkClientFactoryUsingCompletelyNewProviders(
                 "connectAndAllACLUsername", "connectAndAllACLPassword", null, null)
-            .getSolrZkClient(zkServer.getZkAddress(), AbstractZkTestCase.TIMEOUT);
-    try {
+            .getSolrZkClient(zkServer.getZkAddress(), AbstractZkTestCase.TIMEOUT)) {
       AbstractDigestZkACLAndCredentialsProvidersTestBase.doTest(
           zkClient, true, true, true, true, true, true, true, true, true, true);
-    } finally {
-      zkClient.close();
     }
   }
 
   @Test
   public void testReadonlyCredentialsSolrZkClientFactoryUsingCompletelyNewProviders()
       throws Exception {
-    SolrZkClient zkClient =
+    try (SolrZkClient zkClient =
         new SolrZkClientFactoryUsingCompletelyNewProviders(
                 "readonlyACLUsername", "readonlyACLPassword", null, null)
-            .getSolrZkClient(zkServer.getZkAddress(), AbstractZkTestCase.TIMEOUT);
-    try {
+            .getSolrZkClient(zkServer.getZkAddress(), AbstractZkTestCase.TIMEOUT)) {
       AbstractDigestZkACLAndCredentialsProvidersTestBase.doTest(
           zkClient, true, true, false, false, false, false, false, false, false, false);
-    } finally {
-      zkClient.close();
     }
   }
 
@@ -202,16 +199,13 @@ public void testReadonlyCredentialsSolrZkClientFactoryUsingCompletelyNewProvider
           throws Exception {
     useNoCredentials();
 
-    SolrZkClient zkClient =
+    try (SolrZkClient zkClient =
         new SolrZkClientUsingVMParamsProvidersButWithDifferentVMParamsNames(
             new SolrZkClient.Builder()
                 .withUrl(zkServer.getZkAddress())
-                .withTimeout(AbstractZkTestCase.TIMEOUT, TimeUnit.MILLISECONDS));
-    try {
+                .withTimeout(AbstractZkTestCase.TIMEOUT, TimeUnit.MILLISECONDS))) {
       AbstractDigestZkACLAndCredentialsProvidersTestBase.doTest(
           zkClient, false, false, false, false, false, false, false, false, false, false);
-    } finally {
-      zkClient.close();
     }
   }
 
@@ -221,16 +215,13 @@ public void testReadonlyCredentialsSolrZkClientFactoryUsingCompletelyNewProvider
           throws Exception {
     useWrongCredentials();
 
-    SolrZkClient zkClient =
+    try (SolrZkClient zkClient =
         new SolrZkClientUsingVMParamsProvidersButWithDifferentVMParamsNames(
             new SolrZkClient.Builder()
                 .withUrl(zkServer.getZkAddress())
-                .withTimeout(AbstractZkTestCase.TIMEOUT, TimeUnit.MILLISECONDS));
-    try {
+                .withTimeout(AbstractZkTestCase.TIMEOUT, TimeUnit.MILLISECONDS))) {
       AbstractDigestZkACLAndCredentialsProvidersTestBase.doTest(
           zkClient, false, false, false, false, false, false, false, false, false, false);
-    } finally {
-      zkClient.close();
     }
   }
 
@@ -240,16 +231,13 @@ public void testReadonlyCredentialsSolrZkClientFactoryUsingCompletelyNewProvider
           throws Exception {
     useAllCredentials();
 
-    SolrZkClient zkClient =
+    try (SolrZkClient zkClient =
         new SolrZkClientUsingVMParamsProvidersButWithDifferentVMParamsNames(
             new SolrZkClient.Builder()
                 .withUrl(zkServer.getZkAddress())
-                .withTimeout(AbstractZkTestCase.TIMEOUT, TimeUnit.MILLISECONDS));
-    try {
+                .withTimeout(AbstractZkTestCase.TIMEOUT, TimeUnit.MILLISECONDS))) {
       AbstractDigestZkACLAndCredentialsProvidersTestBase.doTest(
           zkClient, true, true, true, true, true, true, true, true, true, true);
-    } finally {
-      zkClient.close();
     }
   }
 
@@ -259,16 +247,13 @@ public void testReadonlyCredentialsSolrZkClientFactoryUsingCompletelyNewProvider
           throws Exception {
     useReadonlyCredentials();
 
-    SolrZkClient zkClient =
+    try (SolrZkClient zkClient =
         new SolrZkClientUsingVMParamsProvidersButWithDifferentVMParamsNames(
             new SolrZkClient.Builder()
                 .withUrl(zkServer.getZkAddress())
-                .withTimeout(AbstractZkTestCase.TIMEOUT, TimeUnit.MILLISECONDS));
-    try {
+                .withTimeout(AbstractZkTestCase.TIMEOUT, TimeUnit.MILLISECONDS))) {
       AbstractDigestZkACLAndCredentialsProvidersTestBase.doTest(
           zkClient, true, true, false, false, false, false, false, false, false, false);
-    } finally {
-      zkClient.close();
     }
   }
 
@@ -291,35 +276,38 @@ public SolrZkClientFactoryUsingCompletelyNewProviders(
       this.digestReadonlyPassword = digestReadonlyPassword;
       zkCredentialsInjector =
           () -> {
-            List<ZkCredentialsInjector.ZkCredential> zkCredentials = new ArrayList<>(2);
             ZkCredentialsInjector.ZkCredential allCreds =
                 new ZkCredentialsInjector.ZkCredential(digestUsername, digestPassword, Perms.ALL);
             ZkCredentialsInjector.ZkCredential readCreds =
                 new ZkCredentialsInjector.ZkCredential(
                     digestReadonlyUsername, digestReadonlyPassword, Perms.READ);
-            zkCredentials.add(allCreds);
-            zkCredentials.add(readCreds);
-            return zkCredentials;
+            return List.of(allCreds, readCreds);
           };
     }
 
     public SolrZkClient getSolrZkClient(String zkServerAddress, int zkClientTimeout) {
+      AuthInfo authInfo = null;
+      if (StrUtils.isNotNullOrEmpty(digestUsername) && StrUtils.isNotNullOrEmpty(digestPassword)) {
+        authInfo =
+            new AuthInfo(
+                "digest", (digestUsername + ":" + digestPassword).getBytes(StandardCharsets.UTF_8));
+      }
+      return getSolrZkClient(zkServerAddress, zkClientTimeout, authInfo);
+    }
 
-      return new SolrZkClient(
-          new SolrZkClient.Builder()
-              .withUrl(zkServerAddress)
-              .withTimeout(zkClientTimeout, TimeUnit.MILLISECONDS)) {
-
-        @Override
-        protected ZkCredentialsProvider createZkCredentialsToAddAutomatically() {
-          return new DigestZkCredentialsProvider(zkCredentialsInjector);
-        }
-
-        @Override
-        public ZkACLProvider createZkACLProvider() {
-          return new DigestZkACLProvider(zkCredentialsInjector);
-        }
-      };
+    public SolrZkClient getSolrZkClient(
+        String zkServerAddress, int zkClientTimeout, AuthInfo authInfo) {
+      final List<AuthInfo> authInfos = new ArrayList<>();
+      if (authInfo != null) {
+        authInfos.add(authInfo);
+      }
+      return new SolrZkClient.Builder()
+          .withUrl(zkServerAddress)
+          .withZkCredentialsProvider(
+              new DigestZkCredentialsProvider(zkCredentialsInjector, authInfos))
+          .withAclProvider(new DigestZkACLProvider(zkCredentialsInjector))
+          .withTimeout(zkClientTimeout, TimeUnit.MILLISECONDS)
+          .build();
     }
   }
 
@@ -349,7 +337,7 @@ protected ZkCredentialsProvider createZkCredentialsToAddAutomatically() {
     }
 
     @Override
-    public ZkACLProvider createZkACLProvider() {
+    public ACLProvider createACLProvider() {
       return new DigestZkACLProvider(vmParamsZkCredentialsInjector);
     }
   }
diff --git a/solr/core/src/test/org/apache/solr/cloud/OverseerTest.java b/solr/core/src/test/org/apache/solr/cloud/OverseerTest.java
index 6c5af8facd9..1350106457d 100644
--- a/solr/core/src/test/org/apache/solr/cloud/OverseerTest.java
+++ b/solr/core/src/test/org/apache/solr/cloud/OverseerTest.java
@@ -58,7 +58,6 @@
 import org.apache.solr.cloud.overseer.NodeMutator;
 import org.apache.solr.cloud.overseer.OverseerAction;
 import org.apache.solr.cloud.overseer.ZkWriteCommand;
-import org.apache.solr.common.AlreadyClosedException;
 import org.apache.solr.common.SolrException;
 import org.apache.solr.common.cloud.ClusterState;
 import org.apache.solr.common.cloud.DocCollection;
@@ -1231,7 +1230,7 @@ public void testShardLeaderChange() throws Exception {
             ZkDistributedQueue q = getOpenOverseer().getStateUpdateQueue();
             q.offer(m);
             break;
-          } catch (SolrException | KeeperException | AlreadyClosedException e) {
+          } catch (SolrException | KeeperException | IllegalStateException e) {
             log.error("error updating state", e);
           }
         }
@@ -1249,7 +1248,7 @@ public void testShardLeaderChange() throws Exception {
                 true,
                 getOpenOverseer());
             break;
-          } catch (SolrException | KeeperException | AlreadyClosedException e) {
+          } catch (SolrException | KeeperException | IllegalStateException e) {
             log.error("error publishing state", e);
           }
         }
@@ -1274,7 +1273,7 @@ public void testShardLeaderChange() throws Exception {
                 true,
                 getOpenOverseer());
             break;
-          } catch (SolrException | AlreadyClosedException e) {
+          } catch (SolrException | IllegalStateException e) {
             log.error("error publishing state", e);
           }
         }
@@ -1294,7 +1293,7 @@ public void testShardLeaderChange() throws Exception {
                 true,
                 getOpenOverseer());
             break;
-          } catch (SolrException | AlreadyClosedException e) {
+          } catch (SolrException | IllegalStateException e) {
             log.error("error publishing state", e);
           }
         }
@@ -1314,7 +1313,7 @@ public void testShardLeaderChange() throws Exception {
                 true,
                 getOpenOverseer());
             break;
-          } catch (SolrException | AlreadyClosedException e) {
+          } catch (SolrException | IllegalStateException e) {
             log.error("error publishing state", e);
           }
         }
diff --git a/solr/core/src/test/org/apache/solr/cloud/TestLeaderElectionZkExpiry.java b/solr/core/src/test/org/apache/solr/cloud/TestLeaderElectionZkExpiry.java
index b84d3f49e37..a871fc7176c 100644
--- a/solr/core/src/test/org/apache/solr/cloud/TestLeaderElectionZkExpiry.java
+++ b/solr/core/src/test/org/apache/solr/cloud/TestLeaderElectionZkExpiry.java
@@ -69,7 +69,8 @@ public void testLeaderElectionWithZkExpiry() throws Exception {
               () -> {
                 TimeOut timeout = new TimeOut(10, TimeUnit.SECONDS, TimeSource.NANO_TIME);
                 while (!timeout.hasTimedOut()) {
-                  server.expire(zkController.getZkClient().getZooKeeper().getSessionId());
+                  long sessionId = zkController.getZkClient().getZkSessionId();
+                  server.expire(sessionId);
                   try {
                     timeout.sleep(10);
                   } catch (InterruptedException e) {
diff --git a/solr/core/src/test/org/apache/solr/cloud/TestPullReplicaErrorHandling.java b/solr/core/src/test/org/apache/solr/cloud/TestPullReplicaErrorHandling.java
index 9906d3041f1..f4af183434c 100644
--- a/solr/core/src/test/org/apache/solr/cloud/TestPullReplicaErrorHandling.java
+++ b/solr/core/src/test/org/apache/solr/cloud/TestPullReplicaErrorHandling.java
@@ -51,6 +51,8 @@
 import org.slf4j.LoggerFactory;
 
 @SuppressSSL(bugUrl = "https://issues.apache.org/jira/browse/SOLR-5776")
+// PRS Defaulting is currently not working, so disable for now
+@SolrCloudTestCase.NoPrs
 public class TestPullReplicaErrorHandling extends SolrCloudTestCase {
 
   private static final int REPLICATION_TIMEOUT_SECS = 10;
diff --git a/solr/core/src/test/org/apache/solr/cloud/TestSizeLimitedDistributedMap.java b/solr/core/src/test/org/apache/solr/cloud/TestSizeLimitedDistributedMap.java
index a534e2e2a30..49cecefca05 100644
--- a/solr/core/src/test/org/apache/solr/cloud/TestSizeLimitedDistributedMap.java
+++ b/solr/core/src/test/org/apache/solr/cloud/TestSizeLimitedDistributedMap.java
@@ -91,8 +91,7 @@ public void testConcurrentCleanup() throws Exception {
             .build()) {
       String path = getAndMakeInitialPath(zkClient);
       DistributedMap map =
-          new SizeLimitedDistributedMap(
-              zkClient, path, numResponsesToStore, (element) -> deletedItems.add(element));
+          new SizeLimitedDistributedMap(zkClient, path, numResponsesToStore, deletedItems::add);
       // fill the map to limit first
       for (int i = 0; i < numResponsesToStore; i++) {
         map.put("xyz_" + i, new byte[0]);
diff --git a/solr/core/src/test/org/apache/solr/cloud/ZkSolrClientTest.java b/solr/core/src/test/org/apache/solr/cloud/ZkSolrClientTest.java
index 4503c862d50..2ce44a8b65c 100644
--- a/solr/core/src/test/org/apache/solr/cloud/ZkSolrClientTest.java
+++ b/solr/core/src/test/org/apache/solr/cloud/ZkSolrClientTest.java
@@ -20,20 +20,16 @@
 
 import java.io.IOException;
 import java.nio.file.Path;
-import java.util.Arrays;
 import java.util.HashSet;
-import java.util.List;
 import java.util.Set;
 import java.util.concurrent.CountDownLatch;
 import java.util.concurrent.TimeUnit;
 import java.util.concurrent.atomic.AtomicInteger;
 import org.apache.solr.SolrTestCaseJ4;
 import org.apache.solr.common.cloud.SolrZkClient;
-import org.apache.solr.common.cloud.ZkCmdExecutor;
 import org.apache.solr.common.cloud.ZkMaintenanceUtils;
 import org.apache.zookeeper.CreateMode;
 import org.apache.zookeeper.KeeperException;
-import org.apache.zookeeper.Op;
 import org.apache.zookeeper.WatchedEvent;
 import org.apache.zookeeper.Watcher;
 import org.apache.zookeeper.data.Stat;
@@ -51,8 +47,8 @@ public static void beforeClass() throws Exception {
   @SuppressWarnings({"try"})
   static class ZkConnection implements AutoCloseable {
 
-    private ZkTestServer server = null;
-    private SolrZkClient zkClient = null;
+    private final ZkTestServer server;
+    private final SolrZkClient zkClient;
 
     ZkConnection() throws Exception {
       Path zkDir = createTempDir("zkData");
@@ -91,15 +87,12 @@ public void testConnect() throws Exception {
   @SuppressWarnings({"try"})
   public void testMakeRootNode() throws Exception {
     try (ZkConnection conn = new ZkConnection()) {
-      final SolrZkClient zkClient =
+      try (SolrZkClient zkClient =
           new SolrZkClient.Builder()
               .withUrl(conn.getServer().getZkHost())
               .withTimeout(AbstractZkTestCase.TIMEOUT, TimeUnit.MILLISECONDS)
-              .build();
-      try {
+              .build()) {
         assertTrue(zkClient.exists("/solr", true));
-      } finally {
-        zkClient.close();
       }
     }
   }
@@ -122,7 +115,7 @@ public void testClean() throws Exception {
 
   public void testReconnect() throws Exception {
     Path zkDir = createTempDir("zkData");
-    ZkTestServer server = null;
+    ZkTestServer server;
     server = new ZkTestServer(zkDir);
     server.run();
     try (SolrZkClient zkClient =
@@ -191,7 +184,8 @@ public void run() {
       // simulate session expiration
 
       // one option
-      server.expire(zkClient.getZooKeeper().getSessionId());
+      long sessionId = zkClient.getZkSessionId();
+      server.expire(sessionId);
 
       // another option
       // zkClient.getSolrZooKeeper().getConnection().disconnect();
@@ -223,37 +217,6 @@ public void run() {
     }
   }
 
-  public void testZkCmdExecutor() throws Exception {
-    Path zkDir = createTempDir("zkData");
-    ZkTestServer server = null;
-
-    try {
-      server = new ZkTestServer(zkDir);
-      server.run();
-
-      final int timeout = random().nextInt(10000) + 5000;
-
-      ZkCmdExecutor zkCmdExecutor = new ZkCmdExecutor(timeout);
-      final long start = System.nanoTime();
-      expectThrows(
-          KeeperException.SessionExpiredException.class,
-          () -> {
-            zkCmdExecutor.retryOperation(
-                () -> {
-                  if (System.nanoTime() - start
-                      > TimeUnit.NANOSECONDS.convert(timeout, TimeUnit.MILLISECONDS)) {
-                    throw new KeeperException.SessionExpiredException();
-                  }
-                  throw new KeeperException.ConnectionLossException();
-                });
-          });
-    } finally {
-      if (server != null) {
-        server.shutdown();
-      }
-    }
-  }
-
   @Test
   @SuppressWarnings({"try"})
   public void testMultipleWatchesAsync() throws Exception {
@@ -444,15 +407,9 @@ public void testZkBehavior() throws Exception {
 
       Stat stat = zkClient.exists("/test-node", null, true);
       int cversion = stat.getCversion();
-      List<Op> ops =
-          Arrays.asList(
-              Op.create(
-                  "/test-node/abc",
-                  null,
-                  zkClient.getZkACLProvider().getACLsToAdd("/test-node/abc"),
-                  CreateMode.PERSISTENT),
-              Op.delete("/test-node/abc", -1));
-      zkClient.multi(ops, true);
+      zkClient.multi(
+          op -> op.create().withMode(CreateMode.PERSISTENT).forPath("/test-node/abc", null),
+          op -> op.delete().withVersion(-1).forPath("/test-node/abc"));
       stat = zkClient.exists("/test-node", null, true);
       assertTrue(stat.getCversion() >= cversion + 2);
     } finally {
diff --git a/solr/core/src/test/org/apache/solr/cloud/overseer/ZkStateWriterTest.java b/solr/core/src/test/org/apache/solr/cloud/overseer/ZkStateWriterTest.java
index 7d05566675a..4008f694359 100644
--- a/solr/core/src/test/org/apache/solr/cloud/overseer/ZkStateWriterTest.java
+++ b/solr/core/src/test/org/apache/solr/cloud/overseer/ZkStateWriterTest.java
@@ -472,9 +472,7 @@ public void testSingleExternalCollectionCompressedState() throws Exception {
         writer.writePendingUpdates();
 
         byte[] data =
-            zkClient
-                .getZooKeeper()
-                .getData(ZkStateReader.COLLECTIONS_ZKNODE + "/c1/state.json", null, null);
+            zkClient.getData(ZkStateReader.COLLECTIONS_ZKNODE + "/c1/state.json", null, null, true);
         Map<?, ?> map = (Map<?, ?>) Utils.fromJSON(data);
         assertNotNull(map.get("c1"));
       }
@@ -511,8 +509,9 @@ public void testSingleExternalCollectionCompressedState() throws Exception {
 
         byte[] data =
             zkClient
-                .getZooKeeper()
-                .getData(ZkStateReader.COLLECTIONS_ZKNODE + "/c2/state.json", null, null);
+                .getCuratorFramework()
+                .getData()
+                .forPath(ZkStateReader.COLLECTIONS_ZKNODE + "/c2/state.json");
         assertTrue(compressor.isCompressedBytes(data));
         Map<?, ?> map = (Map<?, ?>) Utils.fromJSON(compressor.decompressBytes(data));
         assertNotNull(map.get("c2"));
diff --git a/solr/licenses/curator-client-LICENSE-ASL.txt b/solr/licenses/curator-LICENSE-ASL.txt
similarity index 100%
rename from solr/licenses/curator-client-LICENSE-ASL.txt
rename to solr/licenses/curator-LICENSE-ASL.txt
diff --git a/solr/licenses/curator-client-NOTICE.txt b/solr/licenses/curator-NOTICE.txt
similarity index 100%
rename from solr/licenses/curator-client-NOTICE.txt
rename to solr/licenses/curator-NOTICE.txt
diff --git a/solr/licenses/curator-client-5.7.0.jar.sha1 b/solr/licenses/curator-client-5.7.0.jar.sha1
deleted file mode 100644
index 7bb1b27f325..00000000000
--- a/solr/licenses/curator-client-5.7.0.jar.sha1
+++ /dev/null
@@ -1 +0,0 @@
-690e470ffe1c4bea8149d8eca9982842a447ffc6
diff --git a/solr/licenses/curator-client-5.7.1.jar.sha1 b/solr/licenses/curator-client-5.7.1.jar.sha1
new file mode 100644
index 00000000000..a1b050796bc
--- /dev/null
+++ b/solr/licenses/curator-client-5.7.1.jar.sha1
@@ -0,0 +1 @@
+a3637b9c1cdc3360e9f266ccaad2ad46f06e1ebb
diff --git a/solr/licenses/curator-framework-5.7.0.jar.sha1 b/solr/licenses/curator-framework-5.7.0.jar.sha1
deleted file mode 100644
index c826bcd8a3b..00000000000
--- a/solr/licenses/curator-framework-5.7.0.jar.sha1
+++ /dev/null
@@ -1 +0,0 @@
-0b3a2c5191cb2aa609cc9bdd19701bdbd53ee627
diff --git a/solr/licenses/curator-framework-5.7.1.jar.sha1 b/solr/licenses/curator-framework-5.7.1.jar.sha1
new file mode 100644
index 00000000000..a64e2cb9b2b
--- /dev/null
+++ b/solr/licenses/curator-framework-5.7.1.jar.sha1
@@ -0,0 +1 @@
+ce5c5be53ec41c9f0a1f5a970e87cdae0f566aa3
diff --git a/solr/licenses/curator-framework-LICENSE-ASL.txt b/solr/licenses/curator-framework-LICENSE-ASL.txt
deleted file mode 100644
index 7a4a3ea2424..00000000000
--- a/solr/licenses/curator-framework-LICENSE-ASL.txt
+++ /dev/null
@@ -1,202 +0,0 @@
-
-                                 Apache License
-                           Version 2.0, January 2004
-                        http://www.apache.org/licenses/
-
-   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
-
-   1. Definitions.
-
-      "License" shall mean the terms and conditions for use, reproduction,
-      and distribution as defined by Sections 1 through 9 of this document.
-
-      "Licensor" shall mean the copyright owner or entity authorized by
-      the copyright owner that is granting the License.
-
-      "Legal Entity" shall mean the union of the acting entity and all
-      other entities that control, are controlled by, or are under common
-      control with that entity. For the purposes of this definition,
-      "control" means (i) the power, direct or indirect, to cause the
-      direction or management of such entity, whether by contract or
-      otherwise, or (ii) ownership of fifty percent (50%) or more of the
-      outstanding shares, or (iii) beneficial ownership of such entity.
-
-      "You" (or "Your") shall mean an individual or Legal Entity
-      exercising permissions granted by this License.
-
-      "Source" form shall mean the preferred form for making modifications,
-      including but not limited to software source code, documentation
-      source, and configuration files.
-
-      "Object" form shall mean any form resulting from mechanical
-      transformation or translation of a Source form, including but
-      not limited to compiled object code, generated documentation,
-      and conversions to other media types.
-
-      "Work" shall mean the work of authorship, whether in Source or
-      Object form, made available under the License, as indicated by a
-      copyright notice that is included in or attached to the work
-      (an example is provided in the Appendix below).
-
-      "Derivative Works" shall mean any work, whether in Source or Object
-      form, that is based on (or derived from) the Work and for which the
-      editorial revisions, annotations, elaborations, or other modifications
-      represent, as a whole, an original work of authorship. For the purposes
-      of this License, Derivative Works shall not include works that remain
-      separable from, or merely link (or bind by name) to the interfaces of,
-      the Work and Derivative Works thereof.
-
-      "Contribution" shall mean any work of authorship, including
-      the original version of the Work and any modifications or additions
-      to that Work or Derivative Works thereof, that is intentionally
-      submitted to Licensor for inclusion in the Work by the copyright owner
-      or by an individual or Legal Entity authorized to submit on behalf of
-      the copyright owner. For the purposes of this definition, "submitted"
-      means any form of electronic, verbal, or written communication sent
-      to the Licensor or its representatives, including but not limited to
-      communication on electronic mailing lists, source code control systems,
-      and issue tracking systems that are managed by, or on behalf of, the
-      Licensor for the purpose of discussing and improving the Work, but
-      excluding communication that is conspicuously marked or otherwise
-      designated in writing by the copyright owner as "Not a Contribution."
-
-      "Contributor" shall mean Licensor and any individual or Legal Entity
-      on behalf of whom a Contribution has been received by Licensor and
-      subsequently incorporated within the Work.
-
-   2. Grant of Copyright License. Subject to the terms and conditions of
-      this License, each Contributor hereby grants to You a perpetual,
-      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
-      copyright license to reproduce, prepare Derivative Works of,
-      publicly display, publicly perform, sublicense, and distribute the
-      Work and such Derivative Works in Source or Object form.
-
-   3. Grant of Patent License. Subject to the terms and conditions of
-      this License, each Contributor hereby grants to You a perpetual,
-      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
-      (except as stated in this section) patent license to make, have made,
-      use, offer to sell, sell, import, and otherwise transfer the Work,
-      where such license applies only to those patent claims licensable
-      by such Contributor that are necessarily infringed by their
-      Contribution(s) alone or by combination of their Contribution(s)
-      with the Work to which such Contribution(s) was submitted. If You
-      institute patent litigation against any entity (including a
-      cross-claim or counterclaim in a lawsuit) alleging that the Work
-      or a Contribution incorporated within the Work constitutes direct
-      or contributory patent infringement, then any patent licenses
-      granted to You under this License for that Work shall terminate
-      as of the date such litigation is filed.
-
-   4. Redistribution. You may reproduce and distribute copies of the
-      Work or Derivative Works thereof in any medium, with or without
-      modifications, and in Source or Object form, provided that You
-      meet the following conditions:
-
-      (a) You must give any other recipients of the Work or
-          Derivative Works a copy of this License; and
-
-      (b) You must cause any modified files to carry prominent notices
-          stating that You changed the files; and
-
-      (c) You must retain, in the Source form of any Derivative Works
-          that You distribute, all copyright, patent, trademark, and
-          attribution notices from the Source form of the Work,
-          excluding those notices that do not pertain to any part of
-          the Derivative Works; and
-
-      (d) If the Work includes a "NOTICE" text file as part of its
-          distribution, then any Derivative Works that You distribute must
-          include a readable copy of the attribution notices contained
-          within such NOTICE file, excluding those notices that do not
-          pertain to any part of the Derivative Works, in at least one
-          of the following places: within a NOTICE text file distributed
-          as part of the Derivative Works; within the Source form or
-          documentation, if provided along with the Derivative Works; or,
-          within a display generated by the Derivative Works, if and
-          wherever such third-party notices normally appear. The contents
-          of the NOTICE file are for informational purposes only and
-          do not modify the License. You may add Your own attribution
-          notices within Derivative Works that You distribute, alongside
-          or as an addendum to the NOTICE text from the Work, provided
-          that such additional attribution notices cannot be construed
-          as modifying the License.
-
-      You may add Your own copyright statement to Your modifications and
-      may provide additional or different license terms and conditions
-      for use, reproduction, or distribution of Your modifications, or
-      for any such Derivative Works as a whole, provided Your use,
-      reproduction, and distribution of the Work otherwise complies with
-      the conditions stated in this License.
-
-   5. Submission of Contributions. Unless You explicitly state otherwise,
-      any Contribution intentionally submitted for inclusion in the Work
-      by You to the Licensor shall be under the terms and conditions of
-      this License, without any additional terms or conditions.
-      Notwithstanding the above, nothing herein shall supersede or modify
-      the terms of any separate license agreement you may have executed
-      with Licensor regarding such Contributions.
-
-   6. Trademarks. This License does not grant permission to use the trade
-      names, trademarks, service marks, or product names of the Licensor,
-      except as required for reasonable and customary use in describing the
-      origin of the Work and reproducing the content of the NOTICE file.
-
-   7. Disclaimer of Warranty. Unless required by applicable law or
-      agreed to in writing, Licensor provides the Work (and each
-      Contributor provides its Contributions) on an "AS IS" BASIS,
-      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
-      implied, including, without limitation, any warranties or conditions
-      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
-      PARTICULAR PURPOSE. You are solely responsible for determining the
-      appropriateness of using or redistributing the Work and assume any
-      risks associated with Your exercise of permissions under this License.
-
-   8. Limitation of Liability. In no event and under no legal theory,
-      whether in tort (including negligence), contract, or otherwise,
-      unless required by applicable law (such as deliberate and grossly
-      negligent acts) or agreed to in writing, shall any Contributor be
-      liable to You for damages, including any direct, indirect, special,
-      incidental, or consequential damages of any character arising as a
-      result of this License or out of the use or inability to use the
-      Work (including but not limited to damages for loss of goodwill,
-      work stoppage, computer failure or malfunction, or any and all
-      other commercial damages or losses), even if such Contributor
-      has been advised of the possibility of such damages.
-
-   9. Accepting Warranty or Additional Liability. While redistributing
-      the Work or Derivative Works thereof, You may choose to offer,
-      and charge a fee for, acceptance of support, warranty, indemnity,
-      or other liability obligations and/or rights consistent with this
-      License. However, in accepting such obligations, You may act only
-      on Your own behalf and on Your sole responsibility, not on behalf
-      of any other Contributor, and only if You agree to indemnify,
-      defend, and hold each Contributor harmless for any liability
-      incurred by, or claims asserted against, such Contributor by reason
-      of your accepting any such warranty or additional liability.
-
-   END OF TERMS AND CONDITIONS
-
-   APPENDIX: How to apply the Apache License to your work.
-
-      To apply the Apache License to your work, attach the following
-      boilerplate notice, with the fields enclosed by brackets "[]"
-      replaced with your own identifying information. (Don't include
-      the brackets!)  The text should be enclosed in the appropriate
-      comment syntax for the file format. We also recommend that a
-      file or class name and description of purpose be included on the
-      same "printed page" as the copyright notice for easier
-      identification within third-party archives.
-
-   Copyright [yyyy] [name of copyright owner]
-
-   Licensed under the Apache License, Version 2.0 (the "License");
-   you may not use this file except in compliance with the License.
-   You may obtain a copy of the License at
-
-       http://www.apache.org/licenses/LICENSE-2.0
-
-   Unless required by applicable law or agreed to in writing, software
-   distributed under the License is distributed on an "AS IS" BASIS,
-   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-   See the License for the specific language governing permissions and
-   limitations under the License.
\ No newline at end of file
diff --git a/solr/licenses/curator-framework-NOTICE.txt b/solr/licenses/curator-framework-NOTICE.txt
deleted file mode 100644
index f568d0fe32e..00000000000
--- a/solr/licenses/curator-framework-NOTICE.txt
+++ /dev/null
@@ -1,5 +0,0 @@
-Apache Curator
-Copyright 2013-2014 The Apache Software Foundation
-
-This product includes software developed at
-The Apache Software Foundation (http://www.apache.org/).
\ No newline at end of file
diff --git a/solr/licenses/curator-recipes-5.7.0.jar.sha1 b/solr/licenses/curator-recipes-5.7.0.jar.sha1
deleted file mode 100644
index 3b84a54317c..00000000000
--- a/solr/licenses/curator-recipes-5.7.0.jar.sha1
+++ /dev/null
@@ -1 +0,0 @@
-6cd1b05889cc4ab8f074366dbd0493c6d3b87b6f
diff --git a/solr/licenses/curator-recipes-5.7.1.jar.sha1 b/solr/licenses/curator-recipes-5.7.1.jar.sha1
new file mode 100644
index 00000000000..fc1cd77aa2c
--- /dev/null
+++ b/solr/licenses/curator-recipes-5.7.1.jar.sha1
@@ -0,0 +1 @@
+39f8871614f0ed157f87f412bed8fd7b32c457ac
diff --git a/solr/licenses/curator-recipes-NOTICE.txt b/solr/licenses/curator-recipes-NOTICE.txt
deleted file mode 100644
index f568d0fe32e..00000000000
--- a/solr/licenses/curator-recipes-NOTICE.txt
+++ /dev/null
@@ -1,5 +0,0 @@
-Apache Curator
-Copyright 2013-2014 The Apache Software Foundation
-
-This product includes software developed at
-The Apache Software Foundation (http://www.apache.org/).
\ No newline at end of file
diff --git a/solr/licenses/curator-test-5.7.1.jar.sha1 b/solr/licenses/curator-test-5.7.1.jar.sha1
new file mode 100644
index 00000000000..7c40b3c04f2
--- /dev/null
+++ b/solr/licenses/curator-test-5.7.1.jar.sha1
@@ -0,0 +1 @@
+743367fe601ba7ae481a1fa610b78790eeec1f7c
diff --git a/solr/licenses/junit-jupiter-api-5.6.2.jar.sha1 b/solr/licenses/junit-jupiter-api-5.6.2.jar.sha1
new file mode 100644
index 00000000000..4c77692c079
--- /dev/null
+++ b/solr/licenses/junit-jupiter-api-5.6.2.jar.sha1
@@ -0,0 +1 @@
+c9ba885abfe975cda123bf6f8f0a69a1b46956d0
diff --git a/solr/licenses/junit-platform-commons-1.6.2.jar.sha1 b/solr/licenses/junit-platform-commons-1.6.2.jar.sha1
new file mode 100644
index 00000000000..bbd0e93baf5
--- /dev/null
+++ b/solr/licenses/junit-platform-commons-1.6.2.jar.sha1
@@ -0,0 +1 @@
+7644a14b329e76b5fe487628b50fb5eab6ba7d26
diff --git a/solr/licenses/opentest4j-1.2.0.jar.sha1 b/solr/licenses/opentest4j-1.2.0.jar.sha1
new file mode 100644
index 00000000000..1d8cede8888
--- /dev/null
+++ b/solr/licenses/opentest4j-1.2.0.jar.sha1
@@ -0,0 +1 @@
+28c11eb91f9b6d8e200631d46e20a7f407f2a046
diff --git a/solr/licenses/curator-recipes-LICENSE-ASL.txt b/solr/licenses/opentest4j-LICENSE-ASL.txt
similarity index 99%
rename from solr/licenses/curator-recipes-LICENSE-ASL.txt
rename to solr/licenses/opentest4j-LICENSE-ASL.txt
index 7a4a3ea2424..8f71f43fee3 100644
--- a/solr/licenses/curator-recipes-LICENSE-ASL.txt
+++ b/solr/licenses/opentest4j-LICENSE-ASL.txt
@@ -1,4 +1,3 @@
-
                                  Apache License
                            Version 2.0, January 2004
                         http://www.apache.org/licenses/
@@ -179,7 +178,7 @@
    APPENDIX: How to apply the Apache License to your work.
 
       To apply the Apache License to your work, attach the following
-      boilerplate notice, with the fields enclosed by brackets "[]"
+      boilerplate notice, with the fields enclosed by brackets "{}"
       replaced with your own identifying information. (Don't include
       the brackets!)  The text should be enclosed in the appropriate
       comment syntax for the file format. We also recommend that a
@@ -187,7 +186,7 @@
       same "printed page" as the copyright notice for easier
       identification within third-party archives.
 
-   Copyright [yyyy] [name of copyright owner]
+   Copyright {yyyy} {name of copyright owner}
 
    Licensed under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.
@@ -199,4 +198,5 @@
    distributed under the License is distributed on an "AS IS" BASIS,
    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    See the License for the specific language governing permissions and
-   limitations under the License.
\ No newline at end of file
+   limitations under the License.
+
diff --git a/solr/licenses/opentest4j-NOTICE.txt b/solr/licenses/opentest4j-NOTICE.txt
new file mode 100644
index 00000000000..5a42e15815d
--- /dev/null
+++ b/solr/licenses/opentest4j-NOTICE.txt
@@ -0,0 +1,13 @@
+Copyright 2015-2023 the original author or authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+     http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
diff --git a/solr/modules/hadoop-auth/build.gradle b/solr/modules/hadoop-auth/build.gradle
index 6fc729461c7..a038264f1f7 100644
--- a/solr/modules/hadoop-auth/build.gradle
+++ b/solr/modules/hadoop-auth/build.gradle
@@ -50,14 +50,15 @@ dependencies {
   implementation('org.apache.zookeeper:zookeeper', {
     exclude group: "org.apache.yetus", module: "audience-annotations"
   })
-  implementation ('org.apache.zookeeper:zookeeper-jute') {
-    exclude group: 'org.apache.yetus', module: 'audience-annotations'
-  }
   // required for instantiating a Zookeeper server (for embedding ZK or running tests)
   runtimeOnly 'org.xerial.snappy:snappy-java'
-  implementation 'org.apache.curator:curator-client'
-  implementation 'org.apache.curator:curator-framework'
-  runtimeOnly 'org.apache.curator:curator-recipes'
+
+  implementation('org.apache.curator:curator-framework', {
+    exclude group: 'org.apache.zookeeper', module: 'zookeeper'
+  })
+  runtimeOnly('org.apache.curator:curator-recipes', {
+    exclude group: 'org.apache.zookeeper', module: 'zookeeper'
+  })
 
   // Hadoop auth framework
   implementation 'org.apache.hadoop:hadoop-annotations'
@@ -104,9 +105,15 @@ dependencies {
   })
 
   // Zookeeper dependency - some tests like HdfsCloudBackupRestore need this
+  testImplementation('org.apache.curator:curator-client', {
+    exclude group: 'org.apache.zookeeper', module: 'zookeeper'
+  })
   testImplementation('org.apache.zookeeper:zookeeper', {
     exclude group: "org.apache.yetus", module: "audience-annotations"
   })
+  testImplementation ('org.apache.zookeeper:zookeeper-jute') {
+    exclude group: 'org.apache.yetus', module: 'audience-annotations'
+  }
   // required for instantiating a Zookeeper server in tests or embedded
   testRuntimeOnly ('org.xerial.snappy:snappy-java')
 }
diff --git a/solr/modules/hadoop-auth/src/java/org/apache/solr/security/hadoop/DelegationTokenKerberosFilter.java b/solr/modules/hadoop-auth/src/java/org/apache/solr/security/hadoop/DelegationTokenKerberosFilter.java
index b0021c889a5..318ad72e7e7 100644
--- a/solr/modules/hadoop-auth/src/java/org/apache/solr/security/hadoop/DelegationTokenKerberosFilter.java
+++ b/solr/modules/hadoop-auth/src/java/org/apache/solr/security/hadoop/DelegationTokenKerberosFilter.java
@@ -20,23 +20,15 @@
 import static org.apache.hadoop.security.token.delegation.ZKDelegationTokenSecretManager.ZK_DTSM_ZNODE_WORKING_PATH_DEAFULT;
 
 import java.io.IOException;
-import java.util.ArrayList;
 import java.util.Enumeration;
-import java.util.List;
 import java.util.Objects;
-import java.util.concurrent.ExecutorService;
 import javax.servlet.FilterChain;
 import javax.servlet.FilterConfig;
 import javax.servlet.ServletException;
 import javax.servlet.ServletRequest;
 import javax.servlet.ServletResponse;
 import javax.servlet.http.HttpServletRequest;
-import org.apache.curator.RetryPolicy;
-import org.apache.curator.framework.AuthInfo;
 import org.apache.curator.framework.CuratorFramework;
-import org.apache.curator.framework.CuratorFrameworkFactory;
-import org.apache.curator.framework.api.ACLProvider;
-import org.apache.curator.retry.ExponentialBackoffRetry;
 import org.apache.hadoop.conf.Configuration;
 import org.apache.hadoop.security.UserGroupInformation;
 import org.apache.hadoop.security.authentication.server.AuthenticationHandler;
@@ -44,13 +36,8 @@
 import org.apache.hadoop.security.token.delegation.web.HttpUserGroupInformation;
 import org.apache.solr.common.cloud.SecurityAwareZkACLProvider;
 import org.apache.solr.common.cloud.SolrZkClient;
-import org.apache.solr.common.cloud.ZkACLProvider;
-import org.apache.solr.common.cloud.ZkCredentialsProvider;
-import org.apache.solr.common.util.ExecutorUtil;
-import org.apache.solr.common.util.SolrNamedThreadFactory;
 import org.apache.zookeeper.CreateMode;
 import org.apache.zookeeper.KeeperException;
-import org.apache.zookeeper.data.ACL;
 
 /**
  * This is an authentication filter based on Hadoop's {@link DelegationTokenAuthenticationFilter}.
@@ -58,9 +45,6 @@
  * reuse the authentication of an end-user or another application.
  */
 public class DelegationTokenKerberosFilter extends DelegationTokenAuthenticationFilter {
-  private ExecutorService curatorSafeServiceExecutor;
-  private CuratorFramework curatorFramework;
-
   @Override
   public void init(FilterConfig conf) throws ServletException {
     if (conf != null && "zookeeper".equals(conf.getInitParameter("signer.secret.provider"))) {
@@ -129,19 +113,6 @@ public void doFilter(ServletRequest servletRequest, ServletResponse servletRespo
     super.doFilter(request, response, filterChainWrapper);
   }
 
-  @Override
-  public void destroy() {
-    super.destroy();
-    if (curatorFramework != null) {
-      curatorFramework.close();
-      curatorFramework = null;
-    }
-    if (curatorSafeServiceExecutor != null) {
-      ExecutorUtil.shutdownNowAndAwaitTermination(curatorSafeServiceExecutor);
-      curatorSafeServiceExecutor = null;
-    }
-  }
-
   @Override
   protected void initializeAuthHandler(String authHandlerClassName, FilterConfig filterConfig)
       throws ServletException {
@@ -183,108 +154,20 @@ private CuratorFramework getCuratorClientInternal(FilterConfig conf, SolrZkClien
 
   protected CuratorFramework getCuratorClient(SolrZkClient zkClient)
       throws InterruptedException, KeeperException {
-    // should we try to build a RetryPolicy off of the ZkController?
-    RetryPolicy retryPolicy = new ExponentialBackoffRetry(1000, 3);
-    if (zkClient == null) {
-      throw new IllegalArgumentException("zkClient required");
-    }
-    String zkHost = zkClient.getZkServerAddress();
-    String zkChroot = zkHost.contains("/") ? zkHost.substring(zkHost.indexOf('/')) : "";
-    String zkNamespace = zkChroot + SecurityAwareZkACLProvider.SECURITY_ZNODE_PATH;
-    zkNamespace = zkNamespace.startsWith("/") ? zkNamespace.substring(1) : zkNamespace;
-    String zkConnectionString =
-        zkHost.contains("/") ? zkHost.substring(0, zkHost.indexOf('/')) : zkHost;
-    SolrZkToCuratorCredentialsACLs curatorToSolrZk = new SolrZkToCuratorCredentialsACLs(zkClient);
-    final int connectionTimeoutMs = 30000; // this value is currently hard coded, see SOLR-7561.
-
     // Create /security znode upfront. Without this, the curator framework creates this directory
     // path
     // without the appropriate ACL configuration. This issue is possibly related to HADOOP-11973
     try {
       zkClient.makePath(
           SecurityAwareZkACLProvider.SECURITY_ZNODE_PATH, CreateMode.PERSISTENT, true);
-    } catch (KeeperException.NodeExistsException ignore) {
+    } catch (KeeperException.NodeExistsException ex) {
+      // ignore?
     }
 
-    curatorSafeServiceExecutor =
-        ExecutorUtil.newMDCAwareSingleThreadExecutor(
-            new SolrNamedThreadFactory("delegationtokenkerberosfilter-curator-safeService"));
-    curatorFramework =
-        CuratorFrameworkFactory.builder()
-            .namespace(zkNamespace)
-            .connectString(zkConnectionString)
-            .retryPolicy(retryPolicy)
-            .aclProvider(curatorToSolrZk.getACLProvider())
-            .authorization(curatorToSolrZk.getAuthInfos())
-            .sessionTimeoutMs(zkClient.getZkClientTimeout())
-            .connectionTimeoutMs(connectionTimeoutMs)
-            .runSafeService(curatorSafeServiceExecutor)
-            .build();
-    curatorFramework.start();
-
-    return curatorFramework;
-  }
-
-  /** Convert Solr Zk Credentials/ACLs to Curator versions */
-  protected static class SolrZkToCuratorCredentialsACLs {
-    private final String zkChroot;
-    private final ACLProvider aclProvider;
-    private final List<AuthInfo> authInfos;
-
-    public SolrZkToCuratorCredentialsACLs(SolrZkClient zkClient) {
-      this.aclProvider = createACLProvider(zkClient);
-      this.authInfos = createAuthInfo(zkClient);
-      String zkHost = zkClient.getZkServerAddress();
-      this.zkChroot = zkHost.contains("/") ? zkHost.substring(zkHost.indexOf('/')) : null;
-    }
+    // Note - Curator complains if the namespace starts with /
+    String namespace = zkClient.getAbsolutePath(SecurityAwareZkACLProvider.SECURITY_ZNODE_PATH);
+    namespace = namespace.startsWith("/") ? namespace.substring(1) : namespace;
 
-    public ACLProvider getACLProvider() {
-      return aclProvider;
-    }
-
-    public List<AuthInfo> getAuthInfos() {
-      return authInfos;
-    }
-
-    private ACLProvider createACLProvider(SolrZkClient zkClient) {
-      final ZkACLProvider zkACLProvider = zkClient.getZkACLProvider();
-      return new ACLProvider() {
-        @Override
-        public List<ACL> getDefaultAcl() {
-          return zkACLProvider.getACLsToAdd(null);
-        }
-
-        @Override
-        public List<ACL> getAclForPath(String path) {
-          List<ACL> acls = null;
-
-          // The logic in SecurityAwareZkACLProvider does not work when
-          // the Solr zkPath is chrooted (e.g. /solr instead of /). This
-          // due to the fact that the getACLsToAdd(..) callback provides
-          // an absolute path (instead of relative path to the chroot) and
-          // the string comparison in SecurityAwareZkACLProvider fails.
-          if (zkACLProvider instanceof SecurityAwareZkACLProvider && zkChroot != null) {
-            acls = zkACLProvider.getACLsToAdd(path.replace(zkChroot, ""));
-          } else {
-            acls = zkACLProvider.getACLsToAdd(path);
-          }
-
-          return acls;
-        }
-      };
-    }
-
-    private List<AuthInfo> createAuthInfo(SolrZkClient zkClient) {
-      List<AuthInfo> ret = new ArrayList<>();
-
-      // In theory the credentials to add could change here if zookeeper hasn't been initialized
-      ZkCredentialsProvider credentialsProvider =
-          zkClient.getZkClientConnectionStrategy().getZkCredentialsToAddAutomatically();
-      for (ZkCredentialsProvider.ZkCredentials zkCredentials :
-          credentialsProvider.getCredentials()) {
-        ret.add(new AuthInfo(zkCredentials.getScheme(), zkCredentials.getAuth()));
-      }
-      return ret;
-    }
+    return zkClient.getCuratorFramework().usingNamespace(namespace);
   }
 }
diff --git a/solr/modules/hadoop-auth/src/java/org/apache/solr/security/hadoop/HadoopAuthFilter.java b/solr/modules/hadoop-auth/src/java/org/apache/solr/security/hadoop/HadoopAuthFilter.java
index f462ad7e682..25bd363d3ad 100644
--- a/solr/modules/hadoop-auth/src/java/org/apache/solr/security/hadoop/HadoopAuthFilter.java
+++ b/solr/modules/hadoop-auth/src/java/org/apache/solr/security/hadoop/HadoopAuthFilter.java
@@ -20,35 +20,22 @@
 import static org.apache.hadoop.security.token.delegation.ZKDelegationTokenSecretManager.ZK_DTSM_ZNODE_WORKING_PATH_DEAFULT;
 
 import java.io.IOException;
-import java.util.ArrayList;
-import java.util.List;
 import java.util.Objects;
-import java.util.concurrent.ExecutorService;
 import javax.servlet.FilterChain;
 import javax.servlet.FilterConfig;
 import javax.servlet.ServletException;
 import javax.servlet.ServletRequest;
 import javax.servlet.ServletResponse;
 import javax.servlet.http.HttpServletRequest;
-import org.apache.curator.RetryPolicy;
-import org.apache.curator.framework.AuthInfo;
 import org.apache.curator.framework.CuratorFramework;
-import org.apache.curator.framework.CuratorFrameworkFactory;
-import org.apache.curator.framework.api.ACLProvider;
-import org.apache.curator.retry.ExponentialBackoffRetry;
 import org.apache.hadoop.security.UserGroupInformation;
 import org.apache.hadoop.security.authentication.server.AuthenticationHandler;
 import org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter;
 import org.apache.hadoop.security.token.delegation.web.HttpUserGroupInformation;
 import org.apache.solr.common.cloud.SecurityAwareZkACLProvider;
 import org.apache.solr.common.cloud.SolrZkClient;
-import org.apache.solr.common.cloud.ZkACLProvider;
-import org.apache.solr.common.cloud.ZkCredentialsProvider;
-import org.apache.solr.common.util.ExecutorUtil;
-import org.apache.solr.common.util.SolrNamedThreadFactory;
 import org.apache.zookeeper.CreateMode;
 import org.apache.zookeeper.KeeperException;
-import org.apache.zookeeper.data.ACL;
 
 /**
  * This is an authentication filter based on Hadoop's {@link DelegationTokenAuthenticationFilter}.
@@ -60,9 +47,6 @@ public class HadoopAuthFilter extends DelegationTokenAuthenticationFilter {
    */
   static final String DELEGATION_TOKEN_ZK_CLIENT = "solr.kerberos.delegation.token.zk.client";
 
-  private ExecutorService curatorSafeServiceExecutor;
-  private CuratorFramework curatorFramework;
-
   @Override
   public void init(FilterConfig conf) throws ServletException {
     if (conf != null && "zookeeper".equals(conf.getInitParameter("signer.secret.provider"))) {
@@ -111,14 +95,6 @@ public void doFilter(ServletRequest servletRequest, ServletResponse servletRespo
   @Override
   public void destroy() {
     super.destroy();
-    if (curatorFramework != null) {
-      curatorFramework.close();
-      curatorFramework = null;
-    }
-    if (curatorSafeServiceExecutor != null) {
-      ExecutorUtil.shutdownNowAndAwaitTermination(curatorSafeServiceExecutor);
-      curatorSafeServiceExecutor = null;
-    }
   }
 
   @Override
@@ -161,109 +137,21 @@ private CuratorFramework getCuratorClientInternal(FilterConfig conf, SolrZkClien
   }
 
   protected CuratorFramework getCuratorClient(SolrZkClient zkClient)
-      throws KeeperException, InterruptedException {
-    // should we try to build a RetryPolicy off of the ZkController?
-    RetryPolicy retryPolicy = new ExponentialBackoffRetry(1000, 3);
-    if (zkClient == null) {
-      throw new IllegalArgumentException("zkClient required");
-    }
-    String zkHost = zkClient.getZkServerAddress();
-    String zkChroot = zkHost.contains("/") ? zkHost.substring(zkHost.indexOf('/')) : "";
-    String zkNamespace = zkChroot + SecurityAwareZkACLProvider.SECURITY_ZNODE_PATH;
-    zkNamespace = zkNamespace.startsWith("/") ? zkNamespace.substring(1) : zkNamespace;
-    String zkConnectionString =
-        zkHost.contains("/") ? zkHost.substring(0, zkHost.indexOf('/')) : zkHost;
-    SolrZkToCuratorCredentialsACLs curatorToSolrZk = new SolrZkToCuratorCredentialsACLs(zkClient);
-    final int connectionTimeoutMs = 30000; // this value is currently hard coded, see SOLR-7561.
-
+      throws InterruptedException, KeeperException {
     // Create /security znode upfront. Without this, the curator framework creates this directory
     // path
     // without the appropriate ACL configuration. This issue is possibly related to HADOOP-11973
     try {
       zkClient.makePath(
           SecurityAwareZkACLProvider.SECURITY_ZNODE_PATH, CreateMode.PERSISTENT, true);
-    } catch (KeeperException.NodeExistsException ignore) {
+    } catch (KeeperException.NodeExistsException ex) {
+      // ignore?
     }
 
-    curatorSafeServiceExecutor =
-        ExecutorUtil.newMDCAwareSingleThreadExecutor(
-            new SolrNamedThreadFactory("hadoopauthfilter-curator-safeService"));
-    curatorFramework =
-        CuratorFrameworkFactory.builder()
-            .namespace(zkNamespace)
-            .connectString(zkConnectionString)
-            .retryPolicy(retryPolicy)
-            .aclProvider(curatorToSolrZk.getACLProvider())
-            .authorization(curatorToSolrZk.getAuthInfos())
-            .sessionTimeoutMs(zkClient.getZkClientTimeout())
-            .connectionTimeoutMs(connectionTimeoutMs)
-            .runSafeService(curatorSafeServiceExecutor)
-            .build();
-    curatorFramework.start();
+    // Note - Curator complains if the namespace starts with /
+    String namespace = zkClient.getAbsolutePath(SecurityAwareZkACLProvider.SECURITY_ZNODE_PATH);
+    namespace = namespace.startsWith("/") ? namespace.substring(1) : namespace;
 
-    return curatorFramework;
-  }
-
-  /** Convert Solr Zk Credentials/ACLs to Curator versions */
-  protected static class SolrZkToCuratorCredentialsACLs {
-    private final String zkChroot;
-    private final ACLProvider aclProvider;
-    private final List<AuthInfo> authInfos;
-
-    public SolrZkToCuratorCredentialsACLs(SolrZkClient zkClient) {
-      this.aclProvider = createACLProvider(zkClient);
-      this.authInfos = createAuthInfo(zkClient);
-      String zkHost = zkClient.getZkServerAddress();
-      this.zkChroot = zkHost.contains("/") ? zkHost.substring(zkHost.indexOf('/')) : null;
-    }
-
-    public ACLProvider getACLProvider() {
-      return aclProvider;
-    }
-
-    public List<AuthInfo> getAuthInfos() {
-      return authInfos;
-    }
-
-    private ACLProvider createACLProvider(SolrZkClient zkClient) {
-      final ZkACLProvider zkACLProvider = zkClient.getZkACLProvider();
-      return new ACLProvider() {
-        @Override
-        public List<ACL> getDefaultAcl() {
-          return zkACLProvider.getACLsToAdd(null);
-        }
-
-        @Override
-        public List<ACL> getAclForPath(String path) {
-          List<ACL> acls = null;
-
-          // The logic in SecurityAwareZkACLProvider does not work when
-          // the Solr zkPath is chrooted (e.g. /solr instead of /). This
-          // due to the fact that the getACLsToAdd(..) callback provides
-          // an absolute path (instead of relative path to the chroot) and
-          // the string comparison in SecurityAwareZkACLProvider fails.
-          if (zkACLProvider instanceof SecurityAwareZkACLProvider && zkChroot != null) {
-            acls = zkACLProvider.getACLsToAdd(path.replace(zkChroot, ""));
-          } else {
-            acls = zkACLProvider.getACLsToAdd(path);
-          }
-
-          return acls;
-        }
-      };
-    }
-
-    private List<AuthInfo> createAuthInfo(SolrZkClient zkClient) {
-      List<AuthInfo> ret = new ArrayList<>();
-
-      // In theory the credentials to add could change here if zookeeper hasn't been initialized
-      ZkCredentialsProvider credentialsProvider =
-          zkClient.getZkClientConnectionStrategy().getZkCredentialsToAddAutomatically();
-      for (ZkCredentialsProvider.ZkCredentials zkCredentials :
-          credentialsProvider.getCredentials()) {
-        ret.add(new AuthInfo(zkCredentials.getScheme(), zkCredentials.getAuth()));
-      }
-      return ret;
-    }
+    return zkClient.getCuratorFramework().usingNamespace(namespace);
   }
 }
diff --git a/solr/modules/hadoop-auth/src/java/org/apache/solr/security/hadoop/HadoopAuthPlugin.java b/solr/modules/hadoop-auth/src/java/org/apache/solr/security/hadoop/HadoopAuthPlugin.java
index f878ef49b2c..b103ee6919e 100644
--- a/solr/modules/hadoop-auth/src/java/org/apache/solr/security/hadoop/HadoopAuthPlugin.java
+++ b/solr/modules/hadoop-auth/src/java/org/apache/solr/security/hadoop/HadoopAuthPlugin.java
@@ -35,6 +35,7 @@
 import javax.servlet.http.HttpServletRequest;
 import javax.servlet.http.HttpServletResponse;
 import org.apache.hadoop.security.authentication.server.AuthenticationFilter;
+import org.apache.hadoop.security.authentication.util.ZKSignerSecretProvider;
 import org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler;
 import org.apache.solr.client.solrj.impl.Krb5HttpClientBuilder;
 import org.apache.solr.cloud.ZkController;
@@ -190,6 +191,7 @@ protected FilterConfig getInitFilterConfig(Map<String, Object> pluginConfig) {
     ZkController controller = coreContainer.getZkController();
     if (controller != null) {
       servletContext.setAttribute(DELEGATION_TOKEN_ZK_CLIENT, controller.getZkClient());
+      params.put(ZKSignerSecretProvider.DISCONNECT_FROM_ZOOKEEPER_ON_SHUTDOWN, "false");
     }
 
     FilterConfig conf =
diff --git a/solr/modules/hadoop-auth/src/java/org/apache/solr/security/hadoop/KerberosPlugin.java b/solr/modules/hadoop-auth/src/java/org/apache/solr/security/hadoop/KerberosPlugin.java
index 673e485667a..66b9b90dee2 100644
--- a/solr/modules/hadoop-auth/src/java/org/apache/solr/security/hadoop/KerberosPlugin.java
+++ b/solr/modules/hadoop-auth/src/java/org/apache/solr/security/hadoop/KerberosPlugin.java
@@ -32,6 +32,7 @@
 import javax.servlet.http.HttpServletRequest;
 import javax.servlet.http.HttpServletResponse;
 import org.apache.hadoop.security.authentication.util.KerberosName;
+import org.apache.hadoop.security.authentication.util.ZKSignerSecretProvider;
 import org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler;
 import org.apache.http.HttpRequest;
 import org.apache.http.protocol.HttpContext;
@@ -158,6 +159,8 @@ protected FilterConfig getInitFilterConfig(
               "/token");
           // ensure krb5 is setup properly before running curator
           getHttpClientBuilder(SolrHttpClientBuilder.create());
+
+          params.put(ZKSignerSecretProvider.DISCONNECT_FROM_ZOOKEEPER_ON_SHUTDOWN, "false");
         }
       } else {
         log.info(
diff --git a/solr/modules/hadoop-auth/src/test/org/apache/solr/security/hadoop/SaslZkACLProviderTest.java b/solr/modules/hadoop-auth/src/test/org/apache/solr/security/hadoop/SaslZkACLProviderTest.java
index fc39eb0aad8..b3601b5e7fd 100644
--- a/solr/modules/hadoop-auth/src/test/org/apache/solr/security/hadoop/SaslZkACLProviderTest.java
+++ b/solr/modules/hadoop-auth/src/test/org/apache/solr/security/hadoop/SaslZkACLProviderTest.java
@@ -24,6 +24,7 @@
 import java.nio.charset.StandardCharsets;
 import java.nio.file.Path;
 import java.util.concurrent.TimeUnit;
+import org.apache.curator.framework.api.ACLProvider;
 import org.apache.lucene.tests.util.QuickPatchThreadsFilter;
 import org.apache.lucene.util.Constants;
 import org.apache.solr.SolrIgnoredThreadsFilter;
@@ -94,7 +95,8 @@ public void setUp() throws Exception {
 
     try (SolrZkClient zkClient =
         new SolrZkClientWithACLs(zkServer.getZkHost(), AbstractZkTestCase.TIMEOUT)) {
-      ZooKeeperSaslClient saslClient = zkClient.getZooKeeper().getSaslClient();
+      ZooKeeperSaslClient saslClient =
+          zkClient.getCuratorFramework().getZookeeperClient().getZooKeeper().getSaslClient();
       assumeFalse("Could not set up ZK with SASL", saslClient.isFailed());
       zkClient.makePath("/solr", false, true);
     } catch (KeeperException e) {
@@ -188,7 +190,7 @@ public SolrZkClientWithACLs(String zkServerAddress, int zkClientTimeout) {
     }
 
     @Override
-    public ZkACLProvider createZkACLProvider() {
+    public ZkACLProvider createACLProvider() {
       return new SaslZkACLProvider();
     }
   }
@@ -204,7 +206,7 @@ public SolrZkClientNoACLs(String zkServerAddress, int zkClientTimeout) {
     }
 
     @Override
-    public ZkACLProvider createZkACLProvider() {
+    public ACLProvider createACLProvider() {
       return new DefaultZkACLProvider();
     }
   }
diff --git a/solr/modules/hadoop-auth/src/test/org/apache/solr/security/hadoop/TestImpersonationWithHadoopAuth.java b/solr/modules/hadoop-auth/src/test/org/apache/solr/security/hadoop/TestImpersonationWithHadoopAuth.java
index 8679b2788ea..c940391b01e 100644
--- a/solr/modules/hadoop-auth/src/test/org/apache/solr/security/hadoop/TestImpersonationWithHadoopAuth.java
+++ b/solr/modules/hadoop-auth/src/test/org/apache/solr/security/hadoop/TestImpersonationWithHadoopAuth.java
@@ -53,8 +53,7 @@ public static void setupClass() throws Exception {
     InetAddress loopback = InetAddress.getLoopbackAddress();
     Path securityJsonPath =
         TEST_PATH().resolve("security").resolve("hadoop_simple_auth_with_delegation.json");
-    String securityJson =
-        new String(Files.readAllBytes(securityJsonPath), Charset.defaultCharset());
+    String securityJson = Files.readString(securityJsonPath, Charset.defaultCharset());
 
     Map<String, Object> securityConfig = (Map<String, Object>) Utils.fromJSONString(securityJson);
     Map<String, Object> authConfig = (Map<String, Object>) securityConfig.get("authentication");
diff --git a/solr/modules/hadoop-auth/src/test/org/apache/solr/security/hadoop/TestSolrCloudWithDelegationTokens.java b/solr/modules/hadoop-auth/src/test/org/apache/solr/security/hadoop/TestSolrCloudWithDelegationTokens.java
index dfd2d58ed5b..7f2528aa7ca 100644
--- a/solr/modules/hadoop-auth/src/test/org/apache/solr/security/hadoop/TestSolrCloudWithDelegationTokens.java
+++ b/solr/modules/hadoop-auth/src/test/org/apache/solr/security/hadoop/TestSolrCloudWithDelegationTokens.java
@@ -78,10 +78,6 @@ public static void startup() throws Exception {
 
   @AfterClass
   public static void shutdown() throws Exception {
-    if (miniCluster != null) {
-      miniCluster.shutdown();
-      miniCluster = null;
-    }
     if (null != solrClientPrimary) {
       solrClientPrimary.close();
       solrClientPrimary = null;
@@ -90,6 +86,11 @@ public static void shutdown() throws Exception {
       solrClientSecondary.close();
       solrClientSecondary = null;
     }
+
+    if (miniCluster != null) {
+      miniCluster.shutdown();
+      miniCluster = null;
+    }
     System.clearProperty("authenticationPlugin");
     System.clearProperty(KerberosPlugin.DELEGATION_TOKEN_ENABLED);
     System.clearProperty("solr.kerberos.cookie.domain");
diff --git a/solr/solrj-zookeeper/build.gradle b/solr/solrj-zookeeper/build.gradle
index 2e03d9fed3c..5c63a90e05c 100644
--- a/solr/solrj-zookeeper/build.gradle
+++ b/solr/solrj-zookeeper/build.gradle
@@ -35,6 +35,13 @@ dependencies {
     // declare dependencies we use even though already declared by solrj-core
     implementation 'org.slf4j:slf4j-api'
 
+
+    implementation('org.apache.curator:curator-client', {
+        exclude group: 'org.apache.zookeeper', module: 'zookeeper'
+    })
+    api('org.apache.curator:curator-framework', {
+        exclude group: 'org.apache.zookeeper', module: 'zookeeper'
+    })
     api('org.apache.zookeeper:zookeeper', {
         exclude group: "org.apache.yetus", module: "audience-annotations"
     })
diff --git a/solr/solrj-zookeeper/src/java/org/apache/solr/client/solrj/cloud/DistribStateManager.java b/solr/solrj-zookeeper/src/java/org/apache/solr/client/solrj/cloud/DistribStateManager.java
index a8ca13e8be7..4c9ed44cf32 100644
--- a/solr/solrj-zookeeper/src/java/org/apache/solr/client/solrj/cloud/DistribStateManager.java
+++ b/solr/solrj-zookeeper/src/java/org/apache/solr/client/solrj/cloud/DistribStateManager.java
@@ -24,13 +24,13 @@
 import java.util.List;
 import java.util.Map;
 import java.util.NoSuchElementException;
+import org.apache.curator.framework.api.transaction.CuratorTransactionResult;
 import org.apache.solr.common.SolrCloseable;
 import org.apache.solr.common.cloud.DocCollection;
 import org.apache.solr.common.cloud.PerReplicaStates;
+import org.apache.solr.common.cloud.SolrZkClient;
 import org.apache.zookeeper.CreateMode;
 import org.apache.zookeeper.KeeperException;
-import org.apache.zookeeper.Op;
-import org.apache.zookeeper.OpResult;
 import org.apache.zookeeper.Watcher;
 
 /** This interface represents a distributed state repository. */
@@ -92,7 +92,7 @@ void setData(String path, byte[] data, int version)
           KeeperException,
           InterruptedException;
 
-  List<OpResult> multi(final Iterable<Op> ops)
+  List<CuratorTransactionResult> multi(final List<SolrZkClient.CuratorOpBuilder> ops)
       throws BadVersionException,
           NoSuchElementException,
           AlreadyExistsException,
diff --git a/solr/solrj-zookeeper/src/java/org/apache/solr/client/solrj/impl/ZkDistribStateManager.java b/solr/solrj-zookeeper/src/java/org/apache/solr/client/solrj/impl/ZkDistribStateManager.java
index 815a5bd7b4f..a71542cfbdc 100644
--- a/solr/solrj-zookeeper/src/java/org/apache/solr/client/solrj/impl/ZkDistribStateManager.java
+++ b/solr/solrj-zookeeper/src/java/org/apache/solr/client/solrj/impl/ZkDistribStateManager.java
@@ -22,6 +22,7 @@
 import java.util.List;
 import java.util.Map;
 import java.util.NoSuchElementException;
+import org.apache.curator.framework.api.transaction.CuratorTransactionResult;
 import org.apache.solr.client.solrj.cloud.AlreadyExistsException;
 import org.apache.solr.client.solrj.cloud.BadVersionException;
 import org.apache.solr.client.solrj.cloud.DistribStateManager;
@@ -34,8 +35,6 @@
 import org.apache.solr.common.util.Utils;
 import org.apache.zookeeper.CreateMode;
 import org.apache.zookeeper.KeeperException;
-import org.apache.zookeeper.Op;
-import org.apache.zookeeper.OpResult;
 import org.apache.zookeeper.Watcher;
 import org.apache.zookeeper.data.Stat;
 
@@ -199,7 +198,7 @@ public void setData(String path, byte[] data, int version)
   }
 
   @Override
-  public List<OpResult> multi(Iterable<Op> ops)
+  public List<CuratorTransactionResult> multi(List<SolrZkClient.CuratorOpBuilder> ops)
       throws BadVersionException,
           AlreadyExistsException,
           NoSuchElementException,
@@ -207,7 +206,7 @@ public List<OpResult> multi(Iterable<Op> ops)
           KeeperException,
           InterruptedException {
     try {
-      return zkClient.multi(ops, true);
+      return zkClient.multi(ops);
     } catch (KeeperException.NoNodeException e) {
       throw new NoSuchElementException(ops.toString());
     } catch (KeeperException.NodeExistsException e) {
diff --git a/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/BeforeReconnect.java b/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/BeforeReconnect.java
index 928e606c920..6e028056013 100644
--- a/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/BeforeReconnect.java
+++ b/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/BeforeReconnect.java
@@ -16,6 +16,17 @@
  */
 package org.apache.solr.common.cloud;
 
-public interface BeforeReconnect {
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.curator.framework.state.ConnectionState;
+import org.apache.curator.framework.state.ConnectionStateListener;
+
+public interface BeforeReconnect extends ConnectionStateListener {
   public void command();
+
+  @Override
+  default void stateChanged(CuratorFramework client, ConnectionState newState) {
+    if (newState == ConnectionState.LOST || newState == ConnectionState.SUSPENDED) {
+      command();
+    }
+  }
 }
diff --git a/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/ConnectionManager.java b/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/ConnectionManager.java
deleted file mode 100644
index ba527b1eef8..00000000000
--- a/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/ConnectionManager.java
+++ /dev/null
@@ -1,318 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements.  See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License.  You may obtain a copy of the License at
- *
- *     http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.solr.common.cloud;
-
-import static org.apache.zookeeper.Watcher.Event.KeeperState.AuthFailed;
-import static org.apache.zookeeper.Watcher.Event.KeeperState.Disconnected;
-import static org.apache.zookeeper.Watcher.Event.KeeperState.Expired;
-
-import java.lang.invoke.MethodHandles;
-import java.util.concurrent.TimeUnit;
-import java.util.concurrent.TimeoutException;
-import org.apache.solr.common.SolrException;
-import org.apache.zookeeper.WatchedEvent;
-import org.apache.zookeeper.Watcher;
-import org.apache.zookeeper.Watcher.Event.KeeperState;
-import org.apache.zookeeper.ZooKeeper;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-public class ConnectionManager implements Watcher {
-  private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
-
-  private final String name;
-
-  private volatile boolean connected = false;
-
-  private final ZkClientConnectionStrategy connectionStrategy;
-
-  private final String zkServerAddress;
-
-  private final SolrZkClient client;
-
-  private final OnReconnect onReconnect;
-  private final BeforeReconnect beforeReconnect;
-
-  private volatile boolean isClosed = false;
-
-  // Track the likely expired state
-  private static class LikelyExpiredState {
-    private static LikelyExpiredState NOT_EXPIRED =
-        new LikelyExpiredState(StateType.NOT_EXPIRED, 0);
-    private static LikelyExpiredState EXPIRED = new LikelyExpiredState(StateType.EXPIRED, 0);
-
-    public enum StateType {
-      NOT_EXPIRED, // definitely not expired
-      EXPIRED, // definitely expired
-      TRACKING_TIME // not sure, tracking time of last disconnect
-    }
-
-    private StateType stateType;
-    private long lastDisconnectTime;
-
-    public LikelyExpiredState(StateType stateType, long lastDisconnectTime) {
-      this.stateType = stateType;
-      this.lastDisconnectTime = lastDisconnectTime;
-    }
-
-    public boolean isLikelyExpired(long timeToExpire) {
-      return stateType == StateType.EXPIRED
-          || (stateType == StateType.TRACKING_TIME
-              && (System.nanoTime() - lastDisconnectTime
-                  > TimeUnit.NANOSECONDS.convert(timeToExpire, TimeUnit.MILLISECONDS)));
-    }
-  }
-
-  public interface IsClosed {
-    boolean isClosed();
-  }
-
-  private volatile LikelyExpiredState likelyExpiredState = LikelyExpiredState.EXPIRED;
-
-  private IsClosed isClosedCheck;
-
-  public ConnectionManager(
-      String name,
-      SolrZkClient client,
-      String zkServerAddress,
-      ZkClientConnectionStrategy strat,
-      OnReconnect onConnect,
-      BeforeReconnect beforeReconnect,
-      IsClosed isClosed) {
-    this.name = name;
-    this.client = client;
-    this.connectionStrategy = strat;
-    this.zkServerAddress = zkServerAddress;
-    this.onReconnect = onConnect;
-    this.beforeReconnect = beforeReconnect;
-    this.isClosedCheck = isClosed;
-  }
-
-  private synchronized void connected() {
-    connected = true;
-    likelyExpiredState = LikelyExpiredState.NOT_EXPIRED;
-    notifyAll();
-  }
-
-  private synchronized void disconnected() {
-    connected = false;
-    // record the time we expired unless we are already likely expired
-    if (!likelyExpiredState.isLikelyExpired(0)) {
-      likelyExpiredState =
-          new LikelyExpiredState(LikelyExpiredState.StateType.TRACKING_TIME, System.nanoTime());
-    }
-    notifyAll();
-  }
-
-  @Override
-  public void process(WatchedEvent event) {
-    if (event.getState() == AuthFailed
-        || event.getState() == Disconnected
-        || event.getState() == Expired) {
-      log.warn(
-          "Watcher {} name: {} got event {} path: {} type: {}",
-          this,
-          name,
-          event,
-          event.getPath(),
-          event.getType());
-    } else {
-      if (log.isDebugEnabled()) {
-        log.debug(
-            "Watcher {} name: {} got event {} path: {} type: {}",
-            this,
-            name,
-            event,
-            event.getPath(),
-            event.getType());
-      }
-    }
-
-    if (isClosed()) {
-      log.debug("Client->ZooKeeper status change trigger but we are already closed");
-      return;
-    }
-
-    KeeperState state = event.getState();
-
-    if (state == KeeperState.SyncConnected) {
-      log.info("zkClient has connected");
-      connected();
-      connectionStrategy.connected();
-    } else if (state == Expired) {
-      if (isClosed()) {
-        return;
-      }
-      // we don't call disconnected here, because we know we are expired
-      connected = false;
-      likelyExpiredState = LikelyExpiredState.EXPIRED;
-
-      log.warn(
-          "Our previous ZooKeeper session was expired. Attempting to reconnect to recover relationship with ZooKeeper...");
-
-      if (beforeReconnect != null) {
-        try {
-          beforeReconnect.command();
-        } catch (Exception e) {
-          log.warn("Exception running beforeReconnect command", e);
-        }
-      }
-
-      do {
-        // This loop will break if a valid connection is made. If a connection is not made then it
-        // will repeat and try again to create a new connection.
-        try {
-          connectionStrategy.reconnect(
-              zkServerAddress,
-              client.getZkClientTimeout(),
-              this,
-              new ZkClientConnectionStrategy.ZkUpdate() {
-                @Override
-                public void update(ZooKeeper keeper) {
-                  try {
-                    waitForConnected(Long.MAX_VALUE);
-
-                    try {
-                      client.updateKeeper(keeper);
-                    } catch (InterruptedException e) {
-                      closeKeeper(keeper);
-                      Thread.currentThread().interrupt();
-                      // we must have been asked to stop
-                      throw new RuntimeException(e);
-                    }
-
-                    if (onReconnect != null) {
-                      onReconnect.command();
-                    }
-
-                  } catch (Exception e1) {
-                    // if there was a problem creating the new SolrZooKeeper
-                    // or if we cannot run our reconnect command, close the keeper
-                    // our retry loop will try to create one again
-                    closeKeeper(keeper);
-                    throw new RuntimeException(e1);
-                  }
-                }
-
-                private void closeKeeper(ZooKeeper keeper) {
-                  try {
-                    keeper.close();
-                  } catch (InterruptedException e) {
-                    // Restore the interrupted status
-                    Thread.currentThread().interrupt();
-                    log.error("", e);
-                    throw new ZooKeeperException(SolrException.ErrorCode.SERVER_ERROR, "", e);
-                  }
-                }
-              });
-
-          break;
-
-        } catch (Exception e) {
-          log.error("Could not connect due to error, sleeping for 1s and trying again", e);
-          waitSleep(1000);
-        }
-
-      } while (!isClosed());
-      log.info("zkClient Connected: {}", connected);
-    } else if (state == KeeperState.Disconnected) {
-      log.warn("zkClient has disconnected");
-      disconnected();
-      connectionStrategy.disconnected();
-    } else if (state == KeeperState.AuthFailed) {
-      log.warn("zkClient received AuthFailed");
-    }
-  }
-
-  public synchronized boolean isConnectedAndNotClosed() {
-    return !isClosed() && connected;
-  }
-
-  public synchronized boolean isConnected() {
-    return connected;
-  }
-
-  // we use a volatile rather than sync
-  // to avoid possible deadlock on shutdown
-  public void close() {
-    this.isClosed = true;
-    this.likelyExpiredState = LikelyExpiredState.EXPIRED;
-  }
-
-  private boolean isClosed() {
-    return isClosed || isClosedCheck.isClosed();
-  }
-
-  public boolean isLikelyExpired() {
-    return isClosed()
-        || likelyExpiredState.isLikelyExpired((long) (client.getZkClientTimeout() * 0.90));
-  }
-
-  public synchronized void waitSleep(long waitFor) {
-    try {
-      wait(waitFor);
-    } catch (InterruptedException e) {
-      Thread.currentThread().interrupt();
-    }
-  }
-
-  /**
-   * Wait for an established zookeeper connection
-   *
-   * @param waitForConnection time to wait, in ms
-   */
-  public synchronized void waitForConnected(long waitForConnection) throws TimeoutException {
-    log.info("Waiting up to {}ms for client to connect to ZooKeeper", waitForConnection);
-    long expire =
-        System.nanoTime() + TimeUnit.NANOSECONDS.convert(waitForConnection, TimeUnit.MILLISECONDS);
-    long left = 1;
-    while (!connected && left > 0) {
-      if (isClosed()) {
-        break;
-      }
-      try {
-        wait(500);
-      } catch (InterruptedException e) {
-        Thread.currentThread().interrupt();
-        break;
-      }
-      left = expire - System.nanoTime();
-    }
-    if (!connected) {
-      throw new TimeoutException(
-          "Could not connect to ZooKeeper "
-              + zkServerAddress
-              + " within "
-              + waitForConnection
-              + " ms");
-    }
-    log.info("Client is connected to ZooKeeper");
-  }
-
-  public synchronized void waitForDisconnected(long timeout)
-      throws InterruptedException, TimeoutException {
-    long expire = System.nanoTime() + TimeUnit.NANOSECONDS.convert(timeout, TimeUnit.MILLISECONDS);
-    long left = timeout;
-    while (connected && left > 0) {
-      wait(left);
-      left = expire - System.nanoTime();
-    }
-    if (connected) {
-      throw new TimeoutException("Did not disconnect");
-    }
-  }
-}
diff --git a/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/DefaultConnectionStrategy.java b/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/DefaultConnectionStrategy.java
deleted file mode 100644
index 42d8cbe31e9..00000000000
--- a/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/DefaultConnectionStrategy.java
+++ /dev/null
@@ -1,74 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements.  See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License.  You may obtain a copy of the License at
- *
- *     http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.solr.common.cloud;
-
-import java.io.IOException;
-import java.lang.invoke.MethodHandles;
-import java.util.concurrent.TimeoutException;
-import org.apache.solr.common.AlreadyClosedException;
-import org.apache.zookeeper.Watcher;
-import org.apache.zookeeper.ZooKeeper;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-/** TODO: improve backoff retry impl */
-public class DefaultConnectionStrategy extends ZkClientConnectionStrategy {
-
-  private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
-
-  @Override
-  public void connect(String serverAddress, int timeout, Watcher watcher, ZkUpdate updater)
-      throws IOException, InterruptedException, TimeoutException {
-    ZooKeeper zk = createZooKeeper(serverAddress, timeout, watcher);
-    boolean success = false;
-    try {
-      updater.update(zk);
-      success = true;
-    } finally {
-      if (!success) {
-        zk.close();
-      }
-    }
-  }
-
-  @Override
-  public void reconnect(
-      final String serverAddress,
-      final int zkClientTimeout,
-      final Watcher watcher,
-      final ZkUpdate updater)
-      throws IOException, InterruptedException, TimeoutException {
-    log.warn("Connection expired - starting a new one...");
-    ZooKeeper zk = createZooKeeper(serverAddress, zkClientTimeout, watcher);
-    boolean success = false;
-    try {
-      updater.update(zk);
-      success = true;
-      log.info("Reconnected to ZooKeeper");
-    } catch (AlreadyClosedException e) {
-
-    } finally {
-      if (!success) {
-        try {
-          zk.close();
-        } catch (InterruptedException e) {
-          Thread.currentThread().interrupt();
-        }
-      }
-    }
-  }
-}
diff --git a/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/DefaultZkACLProvider.java b/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/DefaultZkACLProvider.java
index 5b43414da9d..2582d01a0b9 100644
--- a/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/DefaultZkACLProvider.java
+++ b/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/DefaultZkACLProvider.java
@@ -22,7 +22,15 @@
 
 public class DefaultZkACLProvider implements ZkACLProvider {
 
-  private List<ACL> globalACLsToAdd;
+  private volatile List<ACL> globalACLsToAdd;
+
+  public DefaultZkACLProvider() {
+    this(null);
+  }
+
+  public DefaultZkACLProvider(List<ACL> globalACLsToAdd) {
+    this.globalACLsToAdd = globalACLsToAdd;
+  }
 
   @Override
   public List<ACL> getACLsToAdd(String zNodePath) {
@@ -35,6 +43,28 @@ public List<ACL> getACLsToAdd(String zNodePath) {
     return globalACLsToAdd;
   }
 
+  @Override
+  public List<ACL> getAclForPath(String zNodePath) {
+    // In default (simple) implementation use the same set of ACLs for all znodes
+    if (globalACLsToAdd == null) {
+      synchronized (this) {
+        if (globalACLsToAdd == null) globalACLsToAdd = createGlobalACLsToAdd();
+      }
+    }
+    return globalACLsToAdd;
+  }
+
+  @Override
+  public List<ACL> getDefaultAcl() {
+    // In default (simple) implementation use the same set of ACLs for all znodes
+    if (globalACLsToAdd == null) {
+      synchronized (this) {
+        if (globalACLsToAdd == null) globalACLsToAdd = createGlobalACLsToAdd();
+      }
+    }
+    return globalACLsToAdd;
+  }
+
   protected List<ACL> createGlobalACLsToAdd() {
     return ZooDefs.Ids.OPEN_ACL_UNSAFE;
   }
diff --git a/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/DefaultZkCredentialsProvider.java b/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/DefaultZkCredentialsProvider.java
index e855052f6a6..cfdd2098f94 100644
--- a/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/DefaultZkCredentialsProvider.java
+++ b/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/DefaultZkCredentialsProvider.java
@@ -17,19 +17,30 @@
 package org.apache.solr.common.cloud;
 
 import java.util.ArrayList;
-import java.util.Collection;
+import java.util.List;
+import org.apache.curator.framework.AuthInfo;
 
 public class DefaultZkCredentialsProvider implements ZkCredentialsProvider {
 
-  private Collection<ZkCredentials> zkCredentials;
+  private volatile List<AuthInfo> zkCredentials;
   protected ZkCredentialsInjector zkCredentialsInjector;
 
   public DefaultZkCredentialsProvider() {
-    this(new DefaultZkCredentialsInjector());
+    this(new DefaultZkCredentialsInjector(), null);
+  }
+
+  public DefaultZkCredentialsProvider(List<AuthInfo> zkCredentials) {
+    this(new DefaultZkCredentialsInjector(), zkCredentials);
   }
 
   public DefaultZkCredentialsProvider(ZkCredentialsInjector zkCredentialsInjector) {
+    this(zkCredentialsInjector, null);
+  }
+
+  public DefaultZkCredentialsProvider(
+      ZkCredentialsInjector zkCredentialsInjector, List<AuthInfo> zkCredentials) {
     this.zkCredentialsInjector = zkCredentialsInjector;
+    this.zkCredentials = zkCredentials;
   }
 
   @Override
@@ -38,7 +49,7 @@ public void setZkCredentialsInjector(ZkCredentialsInjector zkCredentialsInjector
   }
 
   @Override
-  public Collection<ZkCredentials> getCredentials() {
+  public List<AuthInfo> getCredentials() {
     if (zkCredentials == null) {
       synchronized (this) {
         if (zkCredentials == null) zkCredentials = createCredentials();
@@ -47,7 +58,7 @@ public Collection<ZkCredentials> getCredentials() {
     return zkCredentials;
   }
 
-  protected Collection<ZkCredentials> createCredentials() {
+  protected List<AuthInfo> createCredentials() {
     return new ArrayList<>();
   }
 }
diff --git a/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/DigestZkCredentialsProvider.java b/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/DigestZkCredentialsProvider.java
index cfe30286dc9..03cbbfd0c31 100755
--- a/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/DigestZkCredentialsProvider.java
+++ b/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/DigestZkCredentialsProvider.java
@@ -19,8 +19,8 @@
 import java.lang.invoke.MethodHandles;
 import java.nio.charset.StandardCharsets;
 import java.util.ArrayList;
-import java.util.Collection;
 import java.util.List;
+import org.apache.curator.framework.AuthInfo;
 import org.apache.solr.common.util.StrUtils;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
@@ -40,9 +40,18 @@ public DigestZkCredentialsProvider(ZkCredentialsInjector zkCredentialsInjector)
     super(zkCredentialsInjector);
   }
 
+  public DigestZkCredentialsProvider(List<AuthInfo> zkCredentials) {
+    super(new DefaultZkCredentialsInjector(), zkCredentials);
+  }
+
+  public DigestZkCredentialsProvider(
+      ZkCredentialsInjector zkCredentialsInjector, List<AuthInfo> zkCredentials) {
+    super(zkCredentialsInjector, zkCredentials);
+  }
+
   @Override
-  protected Collection<ZkCredentials> createCredentials() {
-    List<ZkCredentials> result = new ArrayList<>(1);
+  protected List<AuthInfo> createCredentials() {
+    List<AuthInfo> result = new ArrayList<>(1);
     List<ZkCredentialsInjector.ZkCredential> zkCredentials =
         zkCredentialsInjector.getZkCredentials();
     log.debug("createCredentials using zkCredentials: {}", zkCredentials);
@@ -52,7 +61,7 @@ protected Collection<ZkCredentials> createCredentials() {
         if (StrUtils.isNotNullOrEmpty(zkCredential.getUsername())
             && StrUtils.isNotNullOrEmpty(zkCredential.getPassword())) {
           result.add(
-              new ZkCredentials(
+              new AuthInfo(
                   "digest",
                   (zkCredential.getUsername() + ":" + zkCredential.getPassword())
                       .getBytes(StandardCharsets.UTF_8)));
diff --git a/solr/test-framework/src/java/org/apache/solr/cloud/TestConnectionStrategy.java b/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/OnDisconnect.java
similarity index 54%
rename from solr/test-framework/src/java/org/apache/solr/cloud/TestConnectionStrategy.java
rename to solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/OnDisconnect.java
index 88d94faaa85..f33013c5185 100644
--- a/solr/test-framework/src/java/org/apache/solr/cloud/TestConnectionStrategy.java
+++ b/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/OnDisconnect.java
@@ -14,22 +14,19 @@
  * See the License for the specific language governing permissions and
  * limitations under the License.
  */
-package org.apache.solr.cloud;
+package org.apache.solr.common.cloud;
 
-import java.io.IOException;
-import org.apache.solr.common.cloud.DefaultConnectionStrategy;
-import org.apache.zookeeper.TestableZooKeeper;
-import org.apache.zookeeper.Watcher;
-import org.apache.zookeeper.ZooKeeper;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.curator.framework.state.ConnectionState;
+import org.apache.curator.framework.state.ConnectionStateListener;
+
+public interface OnDisconnect extends ConnectionStateListener {
+  public void command();
 
-/**
- * Connection strategy that creates instances of {@link TestableZooKeeper} instead of plain {@link
- * ZooKeeper} objects. Useful for adding pause and disconnect events.
- */
-public class TestConnectionStrategy extends DefaultConnectionStrategy {
   @Override
-  protected ZooKeeper newZooKeeperInstance(
-      String serverAddress, int zkClientTimeout, Watcher watcher) throws IOException {
-    return new TestableZooKeeper(serverAddress, zkClientTimeout, watcher);
+  default void stateChanged(CuratorFramework client, ConnectionState newState) {
+    if (newState == ConnectionState.LOST) {
+      command();
+    }
   }
 }
diff --git a/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/OnReconnect.java b/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/OnReconnect.java
index 19acfda4f6a..1863d797938 100644
--- a/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/OnReconnect.java
+++ b/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/OnReconnect.java
@@ -16,7 +16,10 @@
  */
 package org.apache.solr.common.cloud;
 
-import org.apache.zookeeper.KeeperException.SessionExpiredException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.curator.framework.state.ConnectionState;
+import org.apache.curator.framework.state.ConnectionStateListener;
 
 /**
  * Implementations are expected to implement a correct hashCode and equals method needed to uniquely
@@ -25,6 +28,19 @@
  * org.apache.solr.cloud.ZkController#removeOnReconnectListener(OnReconnect) when it no longer needs
  * to be notified of ZK reconnection events.
  */
-public interface OnReconnect {
-  void command() throws SessionExpiredException;
+public interface OnReconnect extends ConnectionStateListener {
+  void command();
+
+  AtomicBoolean sessionEnded = new AtomicBoolean(false);
+
+  @Override
+  default void stateChanged(CuratorFramework client, ConnectionState newState) {
+    if (ConnectionState.RECONNECTED.equals(newState)) {
+      if (sessionEnded.getAndSet(false)) {
+        command();
+      }
+    } else if (ConnectionState.LOST == newState) {
+      sessionEnded.set(true);
+    }
+  }
 }
diff --git a/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/PerReplicaStatesOps.java b/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/PerReplicaStatesOps.java
index dd0b14fd3e1..85fcef00e83 100644
--- a/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/PerReplicaStatesOps.java
+++ b/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/PerReplicaStatesOps.java
@@ -30,7 +30,6 @@
 import org.apache.solr.common.util.CommonTestInjection;
 import org.apache.zookeeper.CreateMode;
 import org.apache.zookeeper.KeeperException;
-import org.apache.zookeeper.Op;
 import org.apache.zookeeper.data.Stat;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
@@ -104,18 +103,17 @@ private void persist(
       log.debug("Per-replica state being persisted for : '{}', ops: {}", znode, operations);
     }
 
-    List<Op> ops = new ArrayList<>(operations.size());
+    List<SolrZkClient.CuratorOpBuilder> ops = new ArrayList<>(operations.size());
     for (PerReplicaStates.Operation op : operations) {
       // the state of the replica is being updated
       String path = znode + "/" + op.state.asString;
       ops.add(
           op.typ == PerReplicaStates.Operation.Type.ADD
-              ? Op.create(
-                  path, null, zkClient.getZkACLProvider().getACLsToAdd(path), CreateMode.PERSISTENT)
-              : Op.delete(path, -1));
+              ? zkOp -> zkOp.create().withMode(CreateMode.PERSISTENT).forPath(path, null)
+              : zkOp -> zkOp.delete().withVersion(-1).forPath(path));
     }
     try {
-      zkClient.multi(ops, true);
+      zkClient.multi(ops);
     } catch (KeeperException e) {
       log.error("Multi-op exception: {}", zkClient.getChildren(znode, null, true));
       throw e;
diff --git a/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/SaslZkACLProvider.java b/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/SaslZkACLProvider.java
index 2fbfbdd2ef8..ca2ba3bfe0a 100644
--- a/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/SaslZkACLProvider.java
+++ b/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/SaslZkACLProvider.java
@@ -17,7 +17,6 @@
 package org.apache.solr.common.cloud;
 
 import java.util.Arrays;
-import java.util.Collections;
 import java.util.List;
 import org.apache.zookeeper.ZooDefs;
 import org.apache.zookeeper.data.ACL;
@@ -43,6 +42,7 @@ protected List<ACL> createNonSecurityACLsToAdd() {
 
   @Override
   protected List<ACL> createSecurityACLsToAdd() {
-    return Collections.singletonList(new ACL(ZooDefs.Perms.ALL, new Id("sasl", superUser)));
+    // Must be Arrays.asList(), Zookeeper does not allow for immutable list types for ACLs
+    return Arrays.asList(new ACL(ZooDefs.Perms.ALL, new Id("sasl", superUser)));
   }
 }
diff --git a/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/SecurityAwareZkACLProvider.java b/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/SecurityAwareZkACLProvider.java
index 1868f53308a..e0a42c79ebb 100644
--- a/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/SecurityAwareZkACLProvider.java
+++ b/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/SecurityAwareZkACLProvider.java
@@ -17,6 +17,7 @@
 package org.apache.solr.common.cloud;
 
 import java.util.List;
+import org.apache.curator.utils.ZKPaths;
 import org.apache.zookeeper.data.ACL;
 
 /**
@@ -26,16 +27,52 @@
 public abstract class SecurityAwareZkACLProvider implements ZkACLProvider {
   public static final String SECURITY_ZNODE_PATH = "/security";
 
-  private List<ACL> nonSecurityACLsToAdd;
-  private List<ACL> securityACLsToAdd;
+  private volatile List<ACL> nonSecurityACLsToAdd;
+  private volatile List<ACL> securityACLsToAdd;
+
+  private final String securityConfPath;
+  private final String securityZNodePath;
+  private final String securityZNodePathDir;
+
   protected ZkCredentialsInjector zkCredentialsInjector;
 
   public SecurityAwareZkACLProvider() {
     this(new DefaultZkCredentialsInjector());
   }
 
+  public SecurityAwareZkACLProvider(String chroot) {
+    this(new DefaultZkCredentialsInjector(), chroot);
+  }
+
   public SecurityAwareZkACLProvider(ZkCredentialsInjector zkCredentialsInjector) {
+    this(zkCredentialsInjector, null);
+  }
+
+  public SecurityAwareZkACLProvider(ZkCredentialsInjector zkCredentialsInjector, String chroot) {
     this.zkCredentialsInjector = zkCredentialsInjector;
+    if (chroot != null) {
+      this.securityConfPath = ZKPaths.makePath(chroot, ZkStateReader.SOLR_SECURITY_CONF_PATH);
+      this.securityZNodePath = ZKPaths.makePath(chroot, SECURITY_ZNODE_PATH);
+      this.securityZNodePathDir = securityZNodePath + "/";
+    } else {
+      this.securityConfPath = null;
+      this.securityZNodePath = null;
+      this.securityZNodePathDir = null;
+    }
+  }
+
+  public SecurityAwareZkACLProvider withChroot(String chroot) {
+    return new SecurityAwareZkACLProvider(chroot) {
+      @Override
+      protected List<ACL> createNonSecurityACLsToAdd() {
+        return SecurityAwareZkACLProvider.this.createNonSecurityACLsToAdd();
+      }
+
+      @Override
+      protected List<ACL> createSecurityACLsToAdd() {
+        return SecurityAwareZkACLProvider.this.createSecurityACLsToAdd();
+      }
+    };
   }
 
   @Override
@@ -52,11 +89,16 @@ public final List<ACL> getACLsToAdd(String zNodePath) {
     }
   }
 
+  @Override
+  public final List<ACL> getDefaultAcl() {
+    return getNonSecurityACLsToAdd();
+  }
+
   protected boolean isSecurityZNodePath(String zNodePath) {
     return zNodePath != null
-        && (zNodePath.equals(ZkStateReader.SOLR_SECURITY_CONF_PATH)
-            || zNodePath.equals(SECURITY_ZNODE_PATH)
-            || zNodePath.startsWith(SECURITY_ZNODE_PATH + "/"));
+        && (zNodePath.equals(securityConfPath)
+            || zNodePath.equals(securityZNodePath)
+            || (securityZNodePathDir != null && zNodePath.startsWith(securityZNodePathDir)));
   }
 
   /**
diff --git a/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/SolrZkClient.java b/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/SolrZkClient.java
index 044500476de..9ae720cb180 100644
--- a/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/SolrZkClient.java
+++ b/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/SolrZkClient.java
@@ -23,24 +23,37 @@
 import java.nio.charset.StandardCharsets;
 import java.nio.file.Files;
 import java.nio.file.Path;
+import java.util.Arrays;
 import java.util.List;
 import java.util.Locale;
 import java.util.concurrent.ExecutorService;
 import java.util.concurrent.RejectedExecutionException;
 import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
 import java.util.concurrent.atomic.LongAdder;
 import java.util.function.BiFunction;
 import java.util.function.Function;
 import java.util.function.Predicate;
 import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import org.apache.curator.RetryPolicy;
+import org.apache.curator.ensemble.fixed.FixedEnsembleProvider;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.curator.framework.CuratorFrameworkFactory;
+import org.apache.curator.framework.api.ACLProvider;
+import org.apache.curator.framework.api.transaction.CuratorOp;
+import org.apache.curator.framework.api.transaction.CuratorTransactionResult;
+import org.apache.curator.framework.api.transaction.TransactionOp;
+import org.apache.curator.retry.ExponentialBackoffRetry;
+import org.apache.curator.utils.ZKPaths;
 import org.apache.solr.client.solrj.SolrServerException;
 import org.apache.solr.client.solrj.impl.SolrZkClientTimeout;
 import org.apache.solr.common.MapWriter;
 import org.apache.solr.common.SolrException;
 import org.apache.solr.common.annotation.JsonProperty;
-import org.apache.solr.common.cloud.ConnectionManager.IsClosed;
 import org.apache.solr.common.util.Compressor;
 import org.apache.solr.common.util.ExecutorUtil;
+import org.apache.solr.common.util.IOUtils;
 import org.apache.solr.common.util.ObjectReleaseTracker;
 import org.apache.solr.common.util.ReflectMapWriter;
 import org.apache.solr.common.util.SolrNamedThreadFactory;
@@ -48,17 +61,12 @@
 import org.apache.solr.common.util.ZLibCompressor;
 import org.apache.zookeeper.CreateMode;
 import org.apache.zookeeper.KeeperException;
-import org.apache.zookeeper.KeeperException.NoAuthException;
 import org.apache.zookeeper.KeeperException.NoNodeException;
-import org.apache.zookeeper.KeeperException.NodeExistsException;
-import org.apache.zookeeper.Op;
-import org.apache.zookeeper.OpResult;
 import org.apache.zookeeper.WatchedEvent;
 import org.apache.zookeeper.Watcher;
-import org.apache.zookeeper.ZooDefs;
-import org.apache.zookeeper.ZooKeeper;
 import org.apache.zookeeper.data.ACL;
 import org.apache.zookeeper.data.Stat;
+import org.apache.zookeeper.server.quorum.flexible.QuorumVerifier;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
@@ -73,11 +81,8 @@ public class SolrZkClient implements Closeable {
 
   private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
 
-  private ConnectionManager connManager;
-
-  private volatile ZooKeeper keeper;
-
-  private ZkCmdExecutor zkCmdExecutor;
+  private ExecutorService curatorSafeServiceExecutor;
+  private CuratorFramework client;
 
   private final ZkMetrics metrics = new ZkMetrics();
 
@@ -90,14 +95,13 @@ public MapWriter getMetrics() {
 
   private final ExecutorService zkCallbackExecutor =
       ExecutorUtil.newMDCAwareCachedThreadPool(new SolrNamedThreadFactory("zkCallback"));
-  private final ExecutorService zkConnManagerCallbackExecutor =
+  private final ExecutorService zkConnectionListenerCallbackExecutor =
       ExecutorUtil.newMDCAwareSingleThreadExecutor(
-          new SolrNamedThreadFactory("zkConnectionManagerCallback"));
+          new SolrNamedThreadFactory("zkConnectionListenerCallback"));
 
   private volatile boolean isClosed = false;
-  private ZkClientConnectionStrategy zkClientConnectionStrategy;
   private int zkClientTimeout;
-  private ZkACLProvider zkACLProvider;
+  private ACLProvider aclProvider;
   private ZkCredentialsInjector zkCredentialsInjector;
   private String zkServerAddress;
   private SolrClassLoader solrClassLoader;
@@ -113,10 +117,10 @@ public SolrZkClient(Builder builder) {
         builder.zkServerAddress,
         builder.zkClientTimeout,
         builder.zkClientConnectTimeout,
-        builder.connectionStrategy,
+        builder.zkCredentialsProvider,
+        builder.aclProvider,
         builder.onReconnect,
         builder.beforeReconnect,
-        builder.zkACLProvider,
         builder.higherLevelIsClosed,
         builder.minStateByteLenForCompression,
         builder.compressor,
@@ -128,10 +132,10 @@ private SolrZkClient(
       String zkServerAddress,
       int zkClientTimeout,
       int clientConnectTimeout,
-      ZkClientConnectionStrategy strat,
+      ZkCredentialsProvider zkCredentialsProvider,
+      ACLProvider aclProvider,
       final OnReconnect onReconnect,
       BeforeReconnect beforeReconnect,
-      ZkACLProvider zkACLProvider,
       IsClosed higherLevelIsClosed,
       int minStateByteLenForCompression,
       Compressor compressor,
@@ -143,87 +147,91 @@ private SolrZkClient(
       return;
     }
     this.zkServerAddress = zkServerAddress;
-    this.higherLevelIsClosed = higherLevelIsClosed;
-    if (strat == null) {
-      String connectionStrategy = System.getProperty("solr.zookeeper.connectionStrategy");
-      strat =
-          ZkClientConnectionStrategy.forName(connectionStrategy, new DefaultConnectionStrategy());
+    String chroot, zkHost;
+    int chrootIndex = zkServerAddress.indexOf('/');
+    if (chrootIndex == -1) {
+      zkHost = zkServerAddress;
+      chroot = null;
+    } else if (chrootIndex == zkServerAddress.length() - 1) {
+      zkHost = zkServerAddress.substring(0, zkServerAddress.length() - 1);
+      chroot = null;
+    } else {
+      zkHost = zkServerAddress.substring(0, chrootIndex);
+      chroot = zkServerAddress.substring(chrootIndex + 1);
     }
-    this.zkClientConnectionStrategy = strat;
 
+    this.higherLevelIsClosed = higherLevelIsClosed;
     this.solrClassLoader = solrClassLoader;
-    if (!strat.hasZkCredentialsToAddAutomatically()) {
+    if (zkCredentialsProvider == null) {
       zkCredentialsInjector =
           useDefaultCredsAndACLs
               ? createZkCredentialsInjector()
               : new DefaultZkCredentialsInjector();
-      ZkCredentialsProvider zkCredentialsToAddAutomatically =
+      zkCredentialsProvider =
           useDefaultCredsAndACLs
               ? createZkCredentialsToAddAutomatically()
               : new DefaultZkCredentialsProvider();
-      strat.setZkCredentialsToAddAutomatically(zkCredentialsToAddAutomatically);
+    }
+    if (aclProvider == null) {
+      aclProvider = createACLProvider();
+    }
+    if (chroot != null && aclProvider instanceof SecurityAwareZkACLProvider) {
+      this.aclProvider = ((SecurityAwareZkACLProvider) aclProvider).withChroot(chroot);
+    } else {
+      this.aclProvider = aclProvider;
     }
 
     this.zkClientTimeout = zkClientTimeout;
-    // we must retry at least as long as the session timeout
-    zkCmdExecutor = new ZkCmdExecutor(zkClientTimeout, SolrZkClient.this::isClosed);
-    connManager =
-        new ConnectionManager(
-            "ZooKeeperConnection Watcher:" + zkServerAddress,
-            this,
-            zkServerAddress,
-            strat,
-            onReconnect,
-            beforeReconnect,
-            SolrZkClient.this::isClosed);
 
+    curatorSafeServiceExecutor =
+        ExecutorUtil.newMDCAwareSingleThreadExecutor(
+            new SolrNamedThreadFactory("curator-safeService"));
+
+    RetryPolicy retryPolicy = new ExponentialBackoffRetry(1000, 3);
+
+    client =
+        CuratorFrameworkFactory.builder()
+            .ensembleProvider(new FixedEnsembleProvider(zkHost))
+            .namespace(chroot)
+            .sessionTimeoutMs(zkClientTimeout)
+            .connectionTimeoutMs(clientConnectTimeout)
+            .aclProvider(this.aclProvider)
+            .authorization(zkCredentialsProvider.getCredentials())
+            .retryPolicy(retryPolicy)
+            .runSafeService(curatorSafeServiceExecutor)
+            .build();
+    if (onReconnect != null) {
+      client
+          .getConnectionStateListenable()
+          .addListener(onReconnect, zkConnectionListenerCallbackExecutor);
+    }
+    if (beforeReconnect != null) {
+      client
+          .getConnectionStateListenable()
+          .addListener(beforeReconnect, zkConnectionListenerCallbackExecutor);
+    }
+    client.start();
     try {
-      strat.connect(
-          zkServerAddress,
-          zkClientTimeout,
-          wrapWatcher(connManager),
-          zooKeeper -> {
-            ZooKeeper oldKeeper = keeper;
-            keeper = zooKeeper;
-            try {
-              closeKeeper(oldKeeper);
-            } finally {
-              if (isClosed) {
-                // we may have been closed
-                closeKeeper(SolrZkClient.this.keeper);
-              }
-            }
-          });
-    } catch (Exception e) {
-      connManager.close();
-      if (keeper != null) {
-        try {
-          keeper.close();
-        } catch (InterruptedException e1) {
-          Thread.currentThread().interrupt();
-        }
+      if (!client.blockUntilConnected(clientConnectTimeout, TimeUnit.MILLISECONDS)) {
+        throw new TimeoutException(
+            String.format(
+                Locale.ROOT,
+                "Timeout while waiting for Zookeeper Client to connect: %d ms",
+                clientConnectTimeout));
       }
-      throw new SolrException(SolrException.ErrorCode.SERVER_ERROR, e);
-    }
-
-    try {
-      connManager.waitForConnected(clientConnectTimeout);
     } catch (Exception e) {
-      connManager.close();
-      try {
-        keeper.close();
-      } catch (InterruptedException e1) {
+      if (e instanceof InterruptedException) {
         Thread.currentThread().interrupt();
       }
-      zkConnManagerCallbackExecutor.shutdown();
+      client.close();
       throw new SolrException(SolrException.ErrorCode.SERVER_ERROR, e);
     }
+
     assert ObjectReleaseTracker.track(this);
-    if (zkACLProvider == null) {
-      this.zkACLProvider =
-          useDefaultCredsAndACLs ? createZkACLProvider() : new DefaultZkACLProvider();
+    if (aclProvider == null) {
+      this.aclProvider = useDefaultCredsAndACLs ? createACLProvider() : new DefaultZkACLProvider();
     } else {
-      this.zkACLProvider = zkACLProvider;
+      this.aclProvider = aclProvider;
     }
 
     if (compressor == null) {
@@ -234,12 +242,8 @@ private SolrZkClient(
     this.minStateByteLenForCompression = minStateByteLenForCompression;
   }
 
-  public ConnectionManager getConnectionManager() {
-    return connManager;
-  }
-
-  public ZkClientConnectionStrategy getZkClientConnectionStrategy() {
-    return zkClientConnectionStrategy;
+  public CuratorFramework getCuratorFramework() {
+    return client;
   }
 
   public static final String ZK_CRED_PROVIDER_CLASS_NAME_VM_PARAM_NAME = "zkCredentialsProvider";
@@ -273,7 +277,7 @@ protected ZkCredentialsProvider createZkCredentialsToAddAutomatically() {
 
   public static final String ZK_ACL_PROVIDER_CLASS_NAME_VM_PARAM_NAME = "zkACLProvider";
 
-  protected ZkACLProvider createZkACLProvider() {
+  protected ACLProvider createACLProvider() {
     String zkACLProviderClassName = System.getProperty(ZK_ACL_PROVIDER_CLASS_NAME_VM_PARAM_NAME);
     if (StrUtils.isNotNullOrEmpty(zkACLProviderClassName)) {
       try {
@@ -336,20 +340,13 @@ protected ZkCredentialsInjector createZkCredentialsInjector() {
 
   /** Returns true if client is connected */
   public boolean isConnected() {
-    return keeper != null && keeper.getState() == ZooKeeper.States.CONNECTED;
+    return client.getZookeeperClient().isConnected();
   }
 
   public void delete(final String path, final int version, boolean retryOnConnLoss)
       throws InterruptedException, KeeperException {
-    if (retryOnConnLoss) {
-      zkCmdExecutor.retryOperation(
-          () -> {
-            keeper.delete(path, version);
-            return null;
-          });
-    } else {
-      keeper.delete(path, version);
-    }
+    runWithCorrectThrows(
+        "deleting znode", () -> client.delete().withVersion(version).forPath(path));
     metrics.deletes.increment();
   }
 
@@ -381,12 +378,10 @@ public Watcher wrapWatcher(final Watcher watcher) {
    */
   public Stat exists(final String path, final Watcher watcher, boolean retryOnConnLoss)
       throws KeeperException, InterruptedException {
-    Stat result = null;
-    if (retryOnConnLoss) {
-      result = zkCmdExecutor.retryOperation(() -> keeper.exists(path, wrapWatcher(watcher)));
-    } else {
-      result = keeper.exists(path, wrapWatcher(watcher));
-    }
+    Stat result =
+        runWithCorrectThrows(
+            "checking exists",
+            () -> client.checkExists().usingWatcher(wrapWatcher(watcher)).forPath(path));
     metrics.existsChecks.increment();
     return result;
   }
@@ -394,12 +389,8 @@ public Stat exists(final String path, final Watcher watcher, boolean retryOnConn
   /** Returns true if path exists */
   public Boolean exists(final String path, boolean retryOnConnLoss)
       throws KeeperException, InterruptedException {
-    Boolean result = null;
-    if (retryOnConnLoss) {
-      result = zkCmdExecutor.retryOperation(() -> keeper.exists(path, null) != null);
-    } else {
-      result = keeper.exists(path, null) != null;
-    }
+    Boolean result =
+        runWithCorrectThrows("checking exists", () -> client.checkExists().forPath(path) != null);
     metrics.existsChecks.increment();
     return result;
   }
@@ -407,12 +398,10 @@ public Boolean exists(final String path, boolean retryOnConnLoss)
   /** Returns children of the node at the path */
   public List<String> getChildren(final String path, final Watcher watcher, boolean retryOnConnLoss)
       throws KeeperException, InterruptedException {
-    List<String> result = null;
-    if (retryOnConnLoss) {
-      return zkCmdExecutor.retryOperation(() -> keeper.getChildren(path, wrapWatcher(watcher)));
-    } else {
-      result = keeper.getChildren(path, wrapWatcher(watcher));
-    }
+    List<String> result =
+        runWithCorrectThrows(
+            "getting children",
+            () -> client.getChildren().usingWatcher(wrapWatcher(watcher)).forPath(path));
 
     metrics.childFetches.increment();
     if (result != null) {
@@ -425,13 +414,16 @@ public List<String> getChildren(final String path, final Watcher watcher, boolea
   public List<String> getChildren(
       final String path, final Watcher watcher, Stat stat, boolean retryOnConnLoss)
       throws KeeperException, InterruptedException {
-    List<String> result = null;
-    if (retryOnConnLoss) {
-      result =
-          zkCmdExecutor.retryOperation(() -> keeper.getChildren(path, wrapWatcher(watcher), stat));
-    } else {
-      result = keeper.getChildren(path, wrapWatcher(watcher), stat);
-    }
+    List<String> result =
+        runWithCorrectThrows(
+            "getting children",
+            () ->
+                client
+                    .getChildren()
+                    .storingStatIn(stat)
+                    .usingWatcher(wrapWatcher(watcher))
+                    .forPath(path));
+
     metrics.childFetches.increment();
     if (result != null) {
       metrics.cumulativeChildrenFetched.add(result.size());
@@ -443,12 +435,15 @@ public List<String> getChildren(
   public byte[] getData(
       final String path, final Watcher watcher, final Stat stat, boolean retryOnConnLoss)
       throws KeeperException, InterruptedException {
-    byte[] result = null;
-    if (retryOnConnLoss) {
-      result = zkCmdExecutor.retryOperation(() -> keeper.getData(path, wrapWatcher(watcher), stat));
-    } else {
-      result = keeper.getData(path, wrapWatcher(watcher), stat);
-    }
+    byte[] result =
+        runWithCorrectThrows(
+            "getting data",
+            () ->
+                client
+                    .getData()
+                    .storingStatIn(stat)
+                    .usingWatcher(wrapWatcher(watcher))
+                    .forPath(path));
     if (compressor.isCompressedBytes(result)) {
       log.debug("Zookeeper data at path {} is compressed", path);
       try {
@@ -477,17 +472,14 @@ public NodeData getNode(final String path, Watcher watcher, boolean retryOnConnL
   /** Returns node's state */
   public Stat setData(final String path, byte data[], final int version, boolean retryOnConnLoss)
       throws KeeperException, InterruptedException {
-    Stat result = null;
     if (SolrZkClient.shouldCompressData(data, path, minStateByteLenForCompression)) {
       // state.json should be compressed before being put to ZK
       data = compressor.compressBytes(data);
     }
-    if (retryOnConnLoss) {
-      byte[] finalData = data;
-      result = zkCmdExecutor.retryOperation(() -> keeper.setData(path, finalData, version));
-    } else {
-      result = keeper.setData(path, data, version);
-    }
+    final byte[] finalData = data;
+    Stat result =
+        runWithCorrectThrows(
+            "setting data", () -> client.setData().withVersion(version).forPath(path, finalData));
     metrics.writes.increment();
     if (data != null) {
       metrics.bytesWritten.add(data.length);
@@ -535,15 +527,9 @@ public void atomicUpdate(String path, BiFunction<Stat, byte[], byte[]> editor)
   public String create(
       final String path, final byte[] data, final CreateMode createMode, boolean retryOnConnLoss)
       throws KeeperException, InterruptedException {
-    String result = null;
-    if (retryOnConnLoss) {
-      result =
-          zkCmdExecutor.retryOperation(
-              () -> keeper.create(path, data, zkACLProvider.getACLsToAdd(path), createMode));
-    } else {
-      List<ACL> acls = zkACLProvider.getACLsToAdd(path);
-      result = keeper.create(path, data, acls, createMode);
-    }
+    String result =
+        runWithCorrectThrows(
+            "creating znode", () -> client.create().withMode(createMode).forPath(path, data));
     metrics.writes.increment();
     if (data != null) {
       metrics.bytesWritten.add(data.length);
@@ -563,13 +549,15 @@ public String create(
       boolean retryOnConnLoss,
       Stat stat)
       throws KeeperException, InterruptedException {
-    if (retryOnConnLoss) {
-      return zkCmdExecutor.retryOperation(
-          () -> keeper.create(path, data, zkACLProvider.getACLsToAdd(path), createMode, stat));
-    } else {
-      List<ACL> acls = zkACLProvider.getACLsToAdd(path);
-      return keeper.create(path, data, acls, createMode, stat);
+    String result =
+        runWithCorrectThrows(
+            "creating znode",
+            () -> client.create().storingStatIn(stat).withMode(createMode).forPath(path, data));
+    metrics.writes.increment();
+    if (data != null) {
+      metrics.bytesWritten.add(data.length);
     }
+    return result;
   }
 
   /**
@@ -667,8 +655,7 @@ public void makePath(
    *
    * <p>skipPathParts will force the call to fail if the first skipPathParts do not exist already.
    *
-   * <p>Note: retryOnConnLoss is only respected for the final node - nodes before that are always
-   * retried on connection loss.
+   * <p>Note: retryOnConnLoss is no longer respected at all and is ignored
    *
    * <p>Note: if failOnExists == false then we will always overwrite the existing data with the
    * given data
@@ -683,76 +670,77 @@ public void makePath(
       int skipPathParts)
       throws KeeperException, InterruptedException {
     log.debug("makePath: {}", path);
-    metrics.writes.increment();
-
-    boolean retry = true;
-
-    if (path.startsWith("/")) {
-      path = path.substring(1, path.length());
+    var createBuilder = client.create();
+    if (!failOnExists) {
+      createBuilder.orSetData();
     }
+
     if (SolrZkClient.shouldCompressData(data, path, minStateByteLenForCompression)) {
       // state.json should be compressed before being put to ZK
       data = compressor.compressBytes(data);
     }
+
+    metrics.writes.increment();
     if (data != null) {
       metrics.bytesWritten.add(data.length);
     }
 
-    String[] paths = path.split("/");
-    StringBuilder sbPath = new StringBuilder();
-    for (int i = 0; i < paths.length; i++) {
-      String pathPiece = paths[i];
-      sbPath.append("/").append(pathPiece);
-      if (i < skipPathParts) {
-        continue;
+    if (path.startsWith("/")) {
+      path = path.substring(1);
+    }
+    if (skipPathParts > 0) {
+      String[] paths = path.split("/");
+      StringBuilder sbPath = new StringBuilder();
+      for (int i = 0; i < paths.length; i++) {
+        if (i == skipPathParts) {
+          break;
+        }
+        String pathPiece = paths[i];
+        sbPath.append("/").append(pathPiece);
       }
-      byte[] bytes = null;
-      final String currentPath = sbPath.toString();
-
-      CreateMode mode = CreateMode.PERSISTENT;
-      if (i == paths.length - 1) {
-        mode = createMode;
-        bytes = data;
-        if (!retryOnConnLoss) retry = false;
+      String skipPathPartsPath = sbPath.toString();
+      final String finalSkipPathPartsPath;
+      if (!skipPathPartsPath.startsWith("/")) {
+        finalSkipPathPartsPath = "/" + skipPathPartsPath;
+      } else {
+        finalSkipPathPartsPath = skipPathPartsPath;
       }
-      try {
-        if (retry) {
-          final CreateMode finalMode = mode;
-          final byte[] finalBytes = bytes;
-          zkCmdExecutor.retryOperation(
-              () -> {
-                keeper.create(
-                    currentPath, finalBytes, zkACLProvider.getACLsToAdd(currentPath), finalMode);
-                return null;
-              });
-        } else {
-          keeper.create(currentPath, bytes, zkACLProvider.getACLsToAdd(currentPath), mode);
-        }
-      } catch (NoAuthException e) {
-        // in auth cases, we may not have permission for an earlier part of a path, which is fine
-        if (i == paths.length - 1 || !exists(currentPath, retryOnConnLoss)) {
-
-          throw e;
-        }
-      } catch (NodeExistsException e) {
-        if (log.isDebugEnabled()) {
-          log.debug("Node exists: {}", e.getPath());
-        }
-
-        if (!failOnExists && i == paths.length - 1) {
-          // TODO: version ? for now, don't worry about race
-          setData(currentPath, data, -1, retryOnConnLoss);
-          // set new watch
-          exists(currentPath, watcher, retryOnConnLoss);
-          return;
-        }
-
-        // ignore unless it's the last node in the path
-        if (i == paths.length - 1) {
-          throw e;
-        }
+      Stat exists =
+          runWithCorrectThrows(
+              "checking skipPathParts exists",
+              () -> client.checkExists().forPath(finalSkipPathPartsPath));
+      if (exists == null) {
+        throw new KeeperException.NoNodeException(finalSkipPathPartsPath + " didn't already exist");
       }
     }
+
+    final String finalPath;
+    if (!path.startsWith("/")) {
+      finalPath = "/" + path;
+    } else {
+      finalPath = path;
+    }
+
+    final byte[] finalData = data;
+    runWithCorrectThrows(
+        "making path",
+        () -> {
+          // This try/catch can be removed when CURATOR-719 is incorporated
+          try {
+            createBuilder
+                .creatingParentsIfNeeded()
+                .withMode(createMode)
+                .forPath(finalPath, finalData);
+          } catch (KeeperException.NodeExistsException e) {
+            // There can be a race exception when this is called in parallel
+            if (!failOnExists) {
+              setData(finalPath, finalData, true);
+            } else {
+              throw e;
+            }
+          }
+          return client.checkExists().usingWatcher(wrapWatcher(watcher)).forPath(finalPath);
+        });
   }
 
   public void makePath(
@@ -761,6 +749,72 @@ public void makePath(
     makePath(zkPath, null, createMode, watcher, retryOnConnLoss);
   }
 
+  /**
+   * Create a node if it does not exist
+   *
+   * @param path the path at which to create the znode
+   */
+  public void ensureExists(final String path) throws KeeperException, InterruptedException {
+    ensureExists(path, null);
+  }
+
+  /**
+   * Create a node if it does not exist
+   *
+   * @param path the path at which to create the znode
+   * @param data the optional data to set on the znode
+   */
+  public void ensureExists(final String path, final byte[] data)
+      throws KeeperException, InterruptedException {
+    ensureExists(path, data, CreateMode.PERSISTENT);
+  }
+
+  /**
+   * Create a node if it does not exist
+   *
+   * @param path the path at which to create the znode
+   * @param data the optional data to set on the znode
+   * @param createMode the mode with which to create the znode
+   */
+  public void ensureExists(final String path, final byte[] data, CreateMode createMode)
+      throws KeeperException, InterruptedException {
+    ensureExists(path, data, createMode, 0);
+  }
+
+  /**
+   * Create a node if it does not exist
+   *
+   * @param path the path at which to create the znode
+   * @param data the optional data to set on the znode
+   * @param createMode the mode with which to create the znode
+   * @param skipPathParts how many path elements to skip
+   */
+  public void ensureExists(
+      final String path, final byte[] data, CreateMode createMode, int skipPathParts)
+      throws KeeperException, InterruptedException {
+    if (exists(path, true)) {
+      return;
+    }
+    try {
+      if (skipPathParts > 0) {
+        int endingIndex = 0;
+        for (int i = 0; i < skipPathParts && endingIndex >= 0; i++) {
+          endingIndex = path.indexOf('/', endingIndex + 1);
+        }
+        if (endingIndex == -1 || endingIndex == path.length() - 1) {
+          throw new KeeperException.NoNodeException(path);
+        }
+        String startingPath = path.substring(0, endingIndex);
+        if (!exists(startingPath, true)) {
+          throw new KeeperException.NoNodeException(startingPath);
+        }
+      }
+      makePath(path, data, createMode, null, true, true, skipPathParts);
+    } catch (KeeperException.NodeExistsException ignored) {
+      // it's okay if another beats us creating the node
+    }
+  }
+
   /** Write data to ZooKeeper. */
   public Stat setData(String path, byte[] data, boolean retryOnConnLoss)
       throws KeeperException, InterruptedException {
@@ -781,14 +835,44 @@ public Stat setData(String path, Path source, boolean retryOnConnLoss)
     return setData(path, Files.readAllBytes(source), retryOnConnLoss);
   }
 
-  public List<OpResult> multi(final Iterable<Op> ops, boolean retryOnConnLoss)
-      throws InterruptedException, KeeperException {
-    List<OpResult> result = null;
-    if (retryOnConnLoss) {
-      result = zkCmdExecutor.retryOperation(() -> keeper.multi(ops));
-    } else {
-      result = keeper.multi(ops);
+  /**
+   * A function that takes a transaction builder and returns a curator operation. This is used for
+   * users to provide a multi-transaction operation to the client, without needing to use the
+   * underlying Curator client to access the transaction builder.
+   */
+  @FunctionalInterface
+  public interface CuratorOpBuilder {
+    CuratorOp build(TransactionOp startingOp) throws Exception;
+
+    default CuratorOp buildWithoutThrows(TransactionOp startingOp) {
+      try {
+        return build(startingOp);
+      } catch (RuntimeException e) {
+        throw e;
+      } catch (Exception e) {
+        throw new RuntimeException(e);
+      }
     }
+  }
+
+  public List<CuratorTransactionResult> multi(CuratorOpBuilder... ops)
+      throws InterruptedException, KeeperException {
+    return multi(Arrays.asList(ops));
+  }
+
+  public List<CuratorTransactionResult> multi(final List<CuratorOpBuilder> ops)
+      throws InterruptedException, KeeperException {
+    List<CuratorTransactionResult> result =
+        runWithCorrectThrows(
+            "executing multi-transaction",
+            () ->
+                client
+                    .transaction()
+                    .forOperations(
+                        ops.stream()
+                            .map(op -> op.buildWithoutThrows(client.transactionOp()))
+                            .collect(Collectors.toList())));
+
     metrics.multiOps.increment();
     if (result != null) {
       metrics.cumulativeMultiOps.add(result.size());
@@ -802,11 +886,9 @@ public void printLayout(String path, int indent, StringBuilder string)
     byte[] data = getData(path, null, null, true);
     List<String> children = getChildren(path, null, true);
     StringBuilder dent = new StringBuilder();
-    for (int i = 0; i < indent; i++) {
-      dent.append(" ");
-    }
+    dent.append(" ".repeat(Math.max(0, indent)));
     string.append(dent).append(path).append(" (").append(children.size()).append(")").append(NEWL);
-    if (data != null) {
+    if (data != null && data.length > 0) {
       String dataString = new String(data, StandardCharsets.UTF_8);
       if (!path.endsWith(".txt") && !path.endsWith(".xml")) {
         string
@@ -845,8 +927,8 @@ public void close() {
     try {
       closeCallbackExecutor();
     } finally {
-      connManager.close();
-      closeKeeper(keeper);
+      IOUtils.closeQuietly(client);
+      ExecutorUtil.shutdownNowAndAwaitTermination(curatorSafeServiceExecutor);
     }
     assert ObjectReleaseTracker.release(this);
   }
@@ -855,32 +937,24 @@ public boolean isClosed() {
     return isClosed || (higherLevelIsClosed != null && higherLevelIsClosed.isClosed());
   }
 
-  /** Allows package private classes to update volatile ZooKeeper. */
-  void updateKeeper(ZooKeeper keeper) throws InterruptedException {
-    ZooKeeper oldKeeper = this.keeper;
-    this.keeper = keeper;
-    if (oldKeeper != null) {
-      oldKeeper.close();
+  public long getZkSessionId() {
+    if (isConnected()) {
+      try {
+        return client.getZookeeperClient().getZooKeeper().getSessionId();
+      } catch (Exception ignored) {
+      }
     }
-    // we might have been closed already
-    if (isClosed) this.keeper.close();
+    return -1;
   }
 
-  public ZooKeeper getZooKeeper() {
-    return keeper;
-  }
-
-  private void closeKeeper(ZooKeeper keeper) {
-    if (keeper != null) {
+  public int getZkSessionTimeout() {
+    if (isConnected()) {
       try {
-        keeper.close();
-      } catch (InterruptedException e) {
-        // Restore the interrupted status
-        Thread.currentThread().interrupt();
-        log.error("", e);
-        throw new ZooKeeperException(SolrException.ErrorCode.SERVER_ERROR, "", e);
+        return client.getZookeeperClient().getLastNegotiatedSessionTimeoutMs();
+      } catch (Exception ignored) {
       }
     }
+    return 0;
   }
 
   private void closeCallbackExecutor() {
@@ -891,7 +965,7 @@ private void closeCallbackExecutor() {
     }
 
     try {
-      ExecutorUtil.shutdownAndAwaitTermination(zkConnManagerCallbackExecutor);
+      ExecutorUtil.shutdownAndAwaitTermination(zkConnectionListenerCallbackExecutor);
     } catch (Exception e) {
       log.error("Error shutting down zkConnManagerCallbackExecutor", e);
     }
@@ -924,6 +998,20 @@ public String getZkServerAddress() {
     return zkServerAddress;
   }
 
+  /**
+   * @return the ChRoot that this client is using for the ZK cluster
+   */
+  public String getChroot() {
+    return client.getNamespace();
+  }
+
+  /**
+   * @return the given path's absolute path in the ZK cluster
+   */
+  public String getAbsolutePath(String path) {
+    return ZKPaths.fixForNamespace(getChroot(), path);
+  }
+
   /**
    * Gets the raw config node /zookeeper/config as returned by server. Response may look like
    *
@@ -938,25 +1026,17 @@ public String getZkServerAddress() {
    *     string
    */
   public String getConfig() {
-    try {
-      Stat stat = new Stat();
-      keeper.sync(ZooDefs.CONFIG_NODE, null, null);
-      byte[] data = keeper.getConfig(false, stat);
-      if (data == null || data.length == 0) {
-        return "";
-      }
-      return new String(data, StandardCharsets.UTF_8);
-    } catch (NoNodeException nne) {
+    QuorumVerifier currentConfig = client.getCurrentConfig();
+    if (currentConfig == null) {
       log.debug("Zookeeper does not have the /zookeeper/config znode, assuming old ZK version");
       return "";
-    } catch (KeeperException | InterruptedException ex) {
-      throw new SolrException(
-          SolrException.ErrorCode.SERVER_ERROR, "Failed to get config from zookeeper", ex);
+    } else {
+      return currentConfig.toString();
     }
   }
 
-  public ZkACLProvider getZkACLProvider() {
-    return zkACLProvider;
+  public ACLProvider getZkACLProvider() {
+    return aclProvider;
   }
 
   /**
@@ -964,11 +1044,8 @@ public ZkACLProvider getZkACLProvider() {
    */
   public List<ACL> getACL(String path, Stat stat, boolean retryOnConnLoss)
       throws KeeperException, InterruptedException {
-    if (retryOnConnLoss) {
-      return zkCmdExecutor.retryOperation(() -> keeper.getACL(path, stat));
-    } else {
-      return keeper.getACL(path, stat);
-    }
+    return runWithCorrectThrows(
+        "getting acls", () -> client.getACL().storingStatIn(stat).forPath(path));
   }
 
   /**
@@ -976,16 +1053,11 @@ public List<ACL> getACL(String path, Stat stat, boolean retryOnConnLoss)
    *
    * @param path path to set ACL on e.g. /solr/conf/solrconfig.xml
    * @param acls a list of {@link ACL}s to be applied
-   * @param retryOnConnLoss true if the command should be retried on connection loss
    * @return the stat of the node
    */
-  public Stat setACL(String path, List<ACL> acls, boolean retryOnConnLoss)
-      throws InterruptedException, KeeperException {
-    if (retryOnConnLoss) {
-      return zkCmdExecutor.retryOperation(() -> keeper.setACL(path, acls, -1));
-    } else {
-      return keeper.setACL(path, acls, -1);
-    }
+  public Stat setACL(String path, List<ACL> acls) throws InterruptedException, KeeperException {
+    return runWithCorrectThrows(
+        "setting acls", () -> client.setACL().withVersion(-1).withACL(acls).forPath(path));
   }
 
   /**
@@ -1000,14 +1072,33 @@ public void updateACLs(final String root) throws KeeperException, InterruptedExc
         ZkMaintenanceUtils.VISIT_ORDER.VISIT_POST,
         path -> {
           try {
-            setACL(path, getZkACLProvider().getACLsToAdd(path), true);
-            log.debug("Updated ACL on {}", path);
+            runWithCorrectThrows(
+                "updating acls", () -> client.setACL().withACL(null).forPath(path));
           } catch (NoNodeException ignored) {
             // If a node was deleted, don't bother trying to set ACLs on it.
           }
         });
   }
 
+  @FunctionalInterface
+  protected interface SupplierWithException<T> {
+    T get() throws Exception;
+  }
+
+  protected <T> T runWithCorrectThrows(String action, SupplierWithException<T> func)
+      throws KeeperException, InterruptedException {
+    try {
+      return func.get();
+    } catch (KeeperException | RuntimeException e) {
+      throw e;
+    } catch (InterruptedException e) {
+      Thread.currentThread().interrupt();
+      throw e;
+    } catch (Exception e) {
+      throw new RuntimeException("Exception occurred while " + action, e);
+    }
+  }
+
   // Some pass-throughs to allow less code disruption to other classes that use SolrZkClient.
   public void clean(String path) throws InterruptedException, KeeperException {
     ZkMaintenanceUtils.clean(this, path);
@@ -1055,6 +1146,11 @@ public void downloadFromZK(String zkPath, Path dir) throws IOException {
     ZkMaintenanceUtils.downloadFromZK(this, zkPath, dir);
   }
 
+  @FunctionalInterface
+  public interface IsClosed {
+    boolean isClosed();
+  }
+
   /**
    * Watcher wrapper that ensures that heavy implementations of process do not interfere with our
    * ability to react to other watches, but also ensures that two wrappers containing equal watches
@@ -1074,15 +1170,11 @@ private final class ProcessWatchWithExecutor implements Watcher { // see below f
     public void process(final WatchedEvent event) {
       log.debug("Submitting job to respond to event {}", event);
       try {
-        if (watcher instanceof ConnectionManager) {
-          zkConnManagerCallbackExecutor.execute(() -> watcher.process(event));
-        } else {
-          zkCallbackExecutor.execute(
-              () -> {
-                metrics.watchesFired.increment();
-                watcher.process(event);
-              });
-        }
+        zkCallbackExecutor.execute(
+            () -> {
+              metrics.watchesFired.increment();
+              watcher.process(event);
+            });
       } catch (RejectedExecutionException e) {
         // If not a graceful shutdown
         if (!isClosed()) {
@@ -1166,8 +1258,8 @@ public static class Builder {
     public int zkClientConnectTimeout = SolrZkClientTimeout.DEFAULT_ZK_CONNECT_TIMEOUT;
     public OnReconnect onReconnect;
     public BeforeReconnect beforeReconnect;
-    public ZkClientConnectionStrategy connectionStrategy;
-    public ZkACLProvider zkACLProvider;
+    public ZkCredentialsProvider zkCredentialsProvider;
+    public ACLProvider aclProvider;
     public IsClosed higherLevelIsClosed;
     public SolrClassLoader solrClassLoader;
     public boolean useDefaultCredsAndACLs = true;
@@ -1221,18 +1313,18 @@ public Builder withReconnectListener(OnReconnect onReconnect) {
       return this;
     }
 
-    public Builder withConnStrategy(ZkClientConnectionStrategy strat) {
-      this.connectionStrategy = strat;
+    public Builder withBeforeConnect(BeforeReconnect beforeReconnect) {
+      this.beforeReconnect = beforeReconnect;
       return this;
     }
 
-    public Builder withBeforeConnect(BeforeReconnect beforeReconnect) {
-      this.beforeReconnect = beforeReconnect;
+    public Builder withZkCredentialsProvider(ZkCredentialsProvider zkCredentialsProvider) {
+      this.zkCredentialsProvider = zkCredentialsProvider;
       return this;
     }
 
-    public Builder withAclProvider(ZkACLProvider zkACLProvider) {
-      this.zkACLProvider = zkACLProvider;
+    public Builder withAclProvider(ACLProvider aclProvider) {
+      this.aclProvider = aclProvider;
       return this;
     }
 
diff --git a/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/VMParamsSingleSetCredentialsDigestZkCredentialsProvider.java b/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/VMParamsSingleSetCredentialsDigestZkCredentialsProvider.java
index 0674a088616..05c3d88d199 100644
--- a/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/VMParamsSingleSetCredentialsDigestZkCredentialsProvider.java
+++ b/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/VMParamsSingleSetCredentialsDigestZkCredentialsProvider.java
@@ -16,7 +16,8 @@
  */
 package org.apache.solr.common.cloud;
 
-import java.util.Collection;
+import java.util.List;
+import org.apache.curator.framework.AuthInfo;
 
 /**
  * Deprecated in favor of a combination of {@link DigestZkCredentialsProvider} and {@link
@@ -64,7 +65,7 @@ public void setZkCredentialsInjector(ZkCredentialsInjector zkCredentialsInjector
   }
 
   @Override
-  protected Collection<ZkCredentials> createCredentials() {
+  protected List<AuthInfo> createCredentials() {
     return digestZkCredentialsProvider.createCredentials();
   }
 }
diff --git a/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/ZkACLProvider.java b/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/ZkACLProvider.java
index 9c1fab51766..4364d0ba236 100644
--- a/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/ZkACLProvider.java
+++ b/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/ZkACLProvider.java
@@ -17,12 +17,18 @@
 package org.apache.solr.common.cloud;
 
 import java.util.List;
+import org.apache.curator.framework.api.ACLProvider;
 import org.apache.zookeeper.data.ACL;
 
-public interface ZkACLProvider {
+public interface ZkACLProvider extends ACLProvider {
 
   List<ACL> getACLsToAdd(String zNodePath);
 
+  @Override
+  default List<ACL> getAclForPath(String zNodePath) {
+    return getACLsToAdd(zNodePath);
+  }
+
   /**
    * @param zkCredentialsInjector The ZkCredentialsInjector that injects ZK credentials
    */
diff --git a/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/ZkClientConnectionStrategy.java b/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/ZkClientConnectionStrategy.java
deleted file mode 100644
index 8102606a59b..00000000000
--- a/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/ZkClientConnectionStrategy.java
+++ /dev/null
@@ -1,155 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements.  See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License.  You may obtain a copy of the License at
- *
- *     http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.solr.common.cloud;
-
-import java.io.IOException;
-import java.lang.invoke.MethodHandles;
-import java.util.ArrayList;
-import java.util.List;
-import java.util.concurrent.TimeoutException;
-import org.apache.solr.common.cloud.ZkCredentialsProvider.ZkCredentials;
-import org.apache.zookeeper.Watcher;
-import org.apache.zookeeper.ZooKeeper;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-/** */
-public abstract class ZkClientConnectionStrategy {
-  private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
-
-  private volatile ZkCredentialsProvider zkCredentialsToAddAutomatically;
-  private volatile boolean zkCredentialsToAddAutomaticallyUsed;
-
-  private List<DisconnectedListener> disconnectedListeners = new ArrayList<>();
-  private List<ConnectedListener> connectedListeners = new ArrayList<>();
-
-  public abstract void connect(
-      String zkServerAddress, int zkClientTimeout, Watcher watcher, ZkUpdate updater)
-      throws IOException, InterruptedException, TimeoutException;
-
-  public abstract void reconnect(
-      String serverAddress, int zkClientTimeout, Watcher watcher, ZkUpdate updater)
-      throws IOException, InterruptedException, TimeoutException;
-
-  public ZkClientConnectionStrategy() {
-    zkCredentialsToAddAutomaticallyUsed = false;
-  }
-
-  public synchronized void disconnected() {
-    for (DisconnectedListener listener : disconnectedListeners) {
-      try {
-        listener.disconnected();
-      } catch (Exception e) {
-        log.error("Exception on disconnected for listener: {}", listener, e);
-      }
-    }
-  }
-
-  public synchronized void connected() {
-    for (ConnectedListener listener : connectedListeners) {
-      try {
-        listener.connected();
-      } catch (Exception e) {
-        log.error("Exception on connected for listener: {}", listener, e);
-      }
-    }
-  }
-
-  public interface DisconnectedListener {
-    void disconnected();
-  }
-
-  public interface ConnectedListener {
-    void connected();
-  }
-
-  public synchronized void addDisconnectedListener(DisconnectedListener listener) {
-    disconnectedListeners.add(listener);
-  }
-
-  public synchronized void removeDisconnectedListener(DisconnectedListener listener) {
-    disconnectedListeners.remove(listener);
-  }
-
-  public synchronized void addConnectedListener(ConnectedListener listener) {
-    connectedListeners.add(listener);
-  }
-
-  public interface ZkUpdate {
-    void update(ZooKeeper zooKeeper) throws InterruptedException, TimeoutException, IOException;
-  }
-
-  public void setZkCredentialsToAddAutomatically(
-      ZkCredentialsProvider zkCredentialsToAddAutomatically) {
-    if (zkCredentialsToAddAutomaticallyUsed || (zkCredentialsToAddAutomatically == null))
-      throw new RuntimeException(
-          "Cannot change zkCredentialsToAddAutomatically after it has been (connect or reconnect was called) used or to null");
-    this.zkCredentialsToAddAutomatically = zkCredentialsToAddAutomatically;
-  }
-
-  public boolean hasZkCredentialsToAddAutomatically() {
-    return zkCredentialsToAddAutomatically != null;
-  }
-
-  public ZkCredentialsProvider getZkCredentialsToAddAutomatically() {
-    return zkCredentialsToAddAutomatically;
-  }
-
-  protected ZooKeeper createZooKeeper(
-      final String serverAddress, final int zkClientTimeout, final Watcher watcher)
-      throws IOException {
-    ZooKeeper result = newZooKeeperInstance(serverAddress, zkClientTimeout, watcher);
-
-    zkCredentialsToAddAutomaticallyUsed = true;
-    for (ZkCredentials zkCredentials : zkCredentialsToAddAutomatically.getCredentials()) {
-      result.addAuthInfo(zkCredentials.getScheme(), zkCredentials.getAuth());
-    }
-
-    return result;
-  }
-
-  // Override for testing
-  protected ZooKeeper newZooKeeperInstance(
-      final String serverAddress, final int zkClientTimeout, final Watcher watcher)
-      throws IOException {
-    return new ZooKeeper(serverAddress, zkClientTimeout, watcher);
-  }
-
-  /**
-   * Instantiate a new connection strategy for the given class name
-   *
-   * @param name the name of the strategy to use
-   * @return the strategy instance, or null if it could not be loaded
-   */
-  public static ZkClientConnectionStrategy forName(String name, ZkClientConnectionStrategy def) {
-    log.debug("Attempting to load zk connection strategy '{}'", name);
-    if (name == null) {
-      return def;
-    }
-
-    try {
-      // TODO should this use SolrResourceLoader?
-      return Class.forName(name)
-          .asSubclass(ZkClientConnectionStrategy.class)
-          .getConstructor()
-          .newInstance();
-    } catch (Exception e) {
-      log.warn("Exception when loading '{}' ZK connection strategy.", name, e);
-      return def;
-    }
-  }
-}
diff --git a/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/ZkCmdExecutor.java b/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/ZkCmdExecutor.java
deleted file mode 100644
index 01924b3c044..00000000000
--- a/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/ZkCmdExecutor.java
+++ /dev/null
@@ -1,103 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements.  See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License.  You may obtain a copy of the License at
- *
- *     http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.solr.common.cloud;
-
-import java.lang.invoke.MethodHandles;
-import org.apache.solr.common.AlreadyClosedException;
-import org.apache.solr.common.cloud.ConnectionManager.IsClosed;
-import org.apache.zookeeper.KeeperException;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-public class ZkCmdExecutor {
-  private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
-
-  private long retryDelay = 1500L; // 1 second would match timeout, so 500 ms over for padding
-  private int retryCount;
-  private double timeouts;
-  private IsClosed isClosed;
-
-  public ZkCmdExecutor(int timeoutms) {
-    this(timeoutms, null);
-  }
-
-  /**
-   * TODO: At this point, this should probably take a SolrZkClient in its constructor.
-   *
-   * @param timeoutms the client timeout for the ZooKeeper clients that will be used with this
-   *     class.
-   */
-  public ZkCmdExecutor(int timeoutms, IsClosed isClosed) {
-    timeouts = timeoutms / 1000.0;
-    this.retryCount = Math.round(0.5f * ((float) Math.sqrt(8.0f * timeouts + 1.0f) - 1.0f)) + 1;
-    this.isClosed = isClosed;
-  }
-
-  public long getRetryDelay() {
-    return retryDelay;
-  }
-
-  public void setRetryDelay(long retryDelay) {
-    this.retryDelay = retryDelay;
-  }
-
-  /** Perform the given operation, retrying if the connection fails */
-  public <T> T retryOperation(ZkOperation<T> operation)
-      throws KeeperException, InterruptedException {
-    KeeperException exception = null;
-    for (int i = 0; i < retryCount; i++) {
-      try {
-        if (log.isTraceEnabled()) {
-          log.trace("Begin zookeeper operation {}, attempt={}", operation, i);
-        }
-        if (i > 0 && isClosed()) {
-          throw new AlreadyClosedException();
-        }
-        return operation.execute();
-      } catch (KeeperException.ConnectionLossException e) {
-        if (exception == null) {
-          exception = e;
-        }
-        if (Thread.currentThread().isInterrupted()) {
-          Thread.currentThread().interrupt();
-          throw new InterruptedException();
-        }
-        if (i != retryCount - 1) {
-          retryDelay(i);
-        }
-      } finally {
-        if (log.isTraceEnabled()) {
-          log.trace("End zookeeper operation {}", operation);
-        }
-      }
-    }
-    throw exception;
-  }
-
-  private boolean isClosed() {
-    return isClosed != null && isClosed.isClosed();
-  }
-
-  /**
-   * Performs a retry delay if this is not the first attempt
-   *
-   * @param attemptCount the number of the attempts performed so far
-   */
-  protected void retryDelay(int attemptCount) throws InterruptedException {
-    Thread.sleep((attemptCount + 1) * retryDelay);
-  }
-}
diff --git a/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/ZkCredentialsProvider.java b/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/ZkCredentialsProvider.java
index 8614ce6880c..24091258593 100644
--- a/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/ZkCredentialsProvider.java
+++ b/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/ZkCredentialsProvider.java
@@ -16,30 +16,11 @@
  */
 package org.apache.solr.common.cloud;
 
-import java.util.Collection;
+import java.util.List;
+import org.apache.curator.framework.AuthInfo;
 
 public interface ZkCredentialsProvider {
-
-  class ZkCredentials {
-    String scheme;
-    byte[] auth;
-
-    public ZkCredentials(String scheme, byte[] auth) {
-      super();
-      this.scheme = scheme;
-      this.auth = auth;
-    }
-
-    public String getScheme() {
-      return scheme;
-    }
-
-    public byte[] getAuth() {
-      return auth;
-    }
-  }
-
-  Collection<ZkCredentials> getCredentials();
+  List<AuthInfo> getCredentials();
 
   void setZkCredentialsInjector(ZkCredentialsInjector zkCredentialsInjector);
 }
diff --git a/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/ZkStateReader.java b/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/ZkStateReader.java
index 0604f00768f..df638085a2e 100644
--- a/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/ZkStateReader.java
+++ b/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/ZkStateReader.java
@@ -573,13 +573,18 @@ public synchronized void createClusterStateWatchersAndUpdate()
       }
       collectionPropertiesZkStateReader.refreshCollectionProperties();
     } catch (KeeperException.NoNodeException nne) {
+      String noNodePath = nne.getPath();
+      if (noNodePath.length() > zkClient.getCuratorFramework().getNamespace().length()) {
+        noNodePath =
+            noNodePath.substring(zkClient.getCuratorFramework().getNamespace().length() + 1);
+      }
       throw new SolrException(
           ErrorCode.SERVICE_UNAVAILABLE,
           "Cannot connect to cluster at "
               + zkClient.getZkServerAddress()
               + ": cluster not found/not ready."
               + " Expected node '"
-              + nne.getPath()
+              + noNodePath
               + "' does not exist.");
     }
   }
@@ -2032,7 +2037,8 @@ public boolean update() throws KeeperException, InterruptedException {
         log.debug("Checking ZK for most up to date Aliases {}", ALIASES);
       }
       // Call sync() first to ensure the subsequent read (getData) is up-to-date.
-      zkClient.getZooKeeper().sync(ALIASES, null, null);
+      zkClient.runWithCorrectThrows(
+          "syncing aliases", () -> zkClient.getCuratorFramework().sync().forPath(ALIASES));
       return setIfNewer(zkClient.getNode(ALIASES, null, true));
     }
 
diff --git a/solr/solrj-zookeeper/src/test/org/apache/solr/common/cloud/SolrZkClientTest.java b/solr/solrj-zookeeper/src/test/org/apache/solr/common/cloud/SolrZkClientTest.java
index 6e59fe3972a..9a3429520c0 100644
--- a/solr/solrj-zookeeper/src/test/org/apache/solr/common/cloud/SolrZkClientTest.java
+++ b/solr/solrj-zookeeper/src/test/org/apache/solr/common/cloud/SolrZkClientTest.java
@@ -21,13 +21,15 @@
 import java.nio.charset.StandardCharsets;
 import java.nio.file.Path;
 import java.security.NoSuchAlgorithmException;
-import java.util.Collection;
+import java.util.Arrays;
 import java.util.Collections;
 import java.util.List;
 import java.util.Optional;
 import java.util.concurrent.Semaphore;
 import java.util.concurrent.TimeUnit;
 import java.util.concurrent.atomic.AtomicInteger;
+import org.apache.curator.framework.AuthInfo;
+import org.apache.curator.framework.api.ACLProvider;
 import org.apache.solr.client.solrj.impl.CloudSolrClient;
 import org.apache.solr.client.solrj.request.CollectionAdminRequest;
 import org.apache.solr.cloud.AbstractZkTestCase;
@@ -102,18 +104,17 @@ public void setUp() throws Exception {
                 .withUrl(zkServer.getZkAddress())
                 .withTimeout(AbstractZkTestCase.TIMEOUT, TimeUnit.MILLISECONDS)) {
           @Override
-          protected ZkACLProvider createZkACLProvider() {
-            return new DefaultZkACLProvider() {
-              @Override
-              protected List<ACL> createGlobalACLsToAdd() {
-                try {
-                  Id id = new Id(SCHEME, DigestAuthenticationProvider.generateDigest(AUTH));
-                  return Collections.singletonList(new ACL(ZooDefs.Perms.ALL, id));
-                } catch (NoSuchAlgorithmException e) {
-                  throw new RuntimeException(e);
-                }
-              }
-            };
+          protected ACLProvider createACLProvider() {
+            try {
+              // Must be Arrays.asList(), Zookeeper does not allow for immutable list types for ACLs
+              return new DefaultZkACLProvider(
+                  Arrays.asList(
+                      new ACL(
+                          ZooDefs.Perms.ALL,
+                          new Id(SCHEME, DigestAuthenticationProvider.generateDigest(AUTH)))));
+            } catch (NoSuchAlgorithmException e) {
+              throw new RuntimeException(e);
+            }
           }
         };
 
@@ -124,13 +125,9 @@ protected List<ACL> createGlobalACLsToAdd() {
                 .withTimeout(AbstractZkTestCase.TIMEOUT, TimeUnit.MILLISECONDS)) {
           @Override
           protected ZkCredentialsProvider createZkCredentialsToAddAutomatically() {
-            return new DefaultZkCredentialsProvider() {
-              @Override
-              protected Collection<ZkCredentials> createCredentials() {
-                return Collections.singleton(
-                    new ZkCredentials(SCHEME, AUTH.getBytes(StandardCharsets.UTF_8)));
-              }
-            };
+            return new DefaultZkCredentialsProvider(
+                Collections.singletonList(
+                    new AuthInfo(SCHEME, AUTH.getBytes(StandardCharsets.UTF_8))));
           }
         };
   }
@@ -273,7 +270,7 @@ private static boolean canRead(SolrZkClient zkClient, String path)
   public void getConfig() {
     // As the embedded ZK is hardcoded to standalone, there is no way to test actual config data
     // here
-    assertEquals("", defaultClient.getConfig());
+    assertEquals("version=0", defaultClient.getConfig());
   }
 
   @Test
diff --git a/solr/solrj-zookeeper/src/test/org/apache/solr/common/cloud/TestZkConfigSetService.java b/solr/solrj-zookeeper/src/test/org/apache/solr/common/cloud/TestZkConfigSetService.java
index 8a2ff9f84cf..8dc77b154e2 100644
--- a/solr/solrj-zookeeper/src/test/org/apache/solr/common/cloud/TestZkConfigSetService.java
+++ b/solr/solrj-zookeeper/src/test/org/apache/solr/common/cloud/TestZkConfigSetService.java
@@ -22,11 +22,12 @@
 import java.nio.file.Path;
 import java.nio.file.Paths;
 import java.security.NoSuchAlgorithmException;
-import java.util.ArrayList;
-import java.util.Collection;
+import java.util.Arrays;
 import java.util.List;
 import java.util.Properties;
 import java.util.concurrent.TimeUnit;
+import org.apache.curator.framework.AuthInfo;
+import org.apache.curator.framework.api.ACLProvider;
 import org.apache.solr.SolrTestCaseJ4;
 import org.apache.solr.cloud.AbstractZkTestCase;
 import org.apache.solr.cloud.ZkConfigSetService;
@@ -143,7 +144,7 @@ public void testUploadConfig() throws IOException {
   }
 
   @Test
-  public void testUploadWithACL() throws IOException {
+  public void testUploadWithACL() throws IOException, NoSuchAlgorithmException {
 
     zkServer.ensurePathExists("/acl");
 
@@ -152,59 +153,36 @@ public void testUploadWithACL() throws IOException {
     final String writeableUsername = "writeable";
     final String writeablePassword = "writeable";
 
-    ZkACLProvider aclProvider =
-        new DefaultZkACLProvider() {
-          @Override
-          protected List<ACL> createGlobalACLsToAdd() {
-            try {
-              List<ACL> result = new ArrayList<>();
-              result.add(
-                  new ACL(
-                      ZooDefs.Perms.ALL,
-                      new Id(
-                          "digest",
-                          DigestAuthenticationProvider.generateDigest(
-                              writeableUsername + ":" + writeablePassword))));
-              result.add(
-                  new ACL(
-                      ZooDefs.Perms.READ,
-                      new Id(
-                          "digest",
-                          DigestAuthenticationProvider.generateDigest(
-                              readOnlyUsername + ":" + readOnlyPassword))));
-              return result;
-            } catch (NoSuchAlgorithmException e) {
-              throw new RuntimeException(e);
-            }
-          }
-        };
-
-    ZkCredentialsProvider readonly =
-        new DefaultZkCredentialsProvider() {
-          @Override
-          protected Collection<ZkCredentials> createCredentials() {
-            List<ZkCredentials> credentials = new ArrayList<>();
-            credentials.add(
-                new ZkCredentials(
+    // Must be Arrays.asList(), Zookeeper does not allow for immutable list types for ACLs
+    List<ACL> acls =
+        Arrays.asList(
+            new ACL(
+                ZooDefs.Perms.ALL,
+                new Id(
                     "digest",
-                    (readOnlyUsername + ":" + readOnlyPassword).getBytes(StandardCharsets.UTF_8)));
-            return credentials;
-          }
-        };
-
-    ZkCredentialsProvider writeable =
-        new DefaultZkCredentialsProvider() {
-          @Override
-          protected Collection<ZkCredentials> createCredentials() {
-            List<ZkCredentials> credentials = new ArrayList<>();
-            credentials.add(
-                new ZkCredentials(
+                    DigestAuthenticationProvider.generateDigest(
+                        writeableUsername + ":" + writeablePassword))),
+            new ACL(
+                ZooDefs.Perms.READ,
+                new Id(
                     "digest",
-                    (writeableUsername + ":" + writeablePassword)
-                        .getBytes(StandardCharsets.UTF_8)));
-            return credentials;
-          }
-        };
+                    DigestAuthenticationProvider.generateDigest(
+                        readOnlyUsername + ":" + readOnlyPassword))));
+    ACLProvider aclProvider = new DefaultZkACLProvider(acls);
+
+    List<AuthInfo> credentials =
+        List.of(
+            new AuthInfo(
+                "digest",
+                (readOnlyUsername + ":" + readOnlyPassword).getBytes(StandardCharsets.UTF_8)));
+    ZkCredentialsProvider readonly = new DefaultZkCredentialsProvider(credentials);
+
+    List<AuthInfo> writeableCredentials =
+        List.of(
+            new AuthInfo(
+                "digest",
+                (writeableUsername + ":" + writeablePassword).getBytes(StandardCharsets.UTF_8)));
+    ZkCredentialsProvider writeable = new DefaultZkCredentialsProvider(writeableCredentials);
 
     Path configPath = createTempDir("acl-config");
     Files.createFile(configPath.resolve("file1"));
@@ -265,7 +243,7 @@ public void testBootstrapConf() throws IOException, KeeperException, Interrupted
 
   static SolrZkClient buildZkClient(
       String zkAddress,
-      final ZkACLProvider aclProvider,
+      final ACLProvider aclProvider,
       final ZkCredentialsProvider credentialsProvider) {
     return new SolrZkClient(
         new SolrZkClient.Builder().withUrl(zkAddress).withTimeout(10000, TimeUnit.MILLISECONDS)) {
@@ -275,7 +253,7 @@ protected ZkCredentialsProvider createZkCredentialsToAddAutomatically() {
       }
 
       @Override
-      protected ZkACLProvider createZkACLProvider() {
+      protected ACLProvider createACLProvider() {
         return aclProvider;
       }
     };
diff --git a/solr/solrj/src/java/org/apache/solr/client/solrj/impl/SolrZkClientTimeout.java b/solr/solrj/src/java/org/apache/solr/client/solrj/impl/SolrZkClientTimeout.java
index 0a0cfd7fdad..a46f269060b 100644
--- a/solr/solrj/src/java/org/apache/solr/client/solrj/impl/SolrZkClientTimeout.java
+++ b/solr/solrj/src/java/org/apache/solr/client/solrj/impl/SolrZkClientTimeout.java
@@ -18,6 +18,8 @@
 
 public class SolrZkClientTimeout {
   public static final int DEFAULT_ZK_CLIENT_TIMEOUT = Integer.getInteger("zkClientTimeout", 30000);
+  // Make this overridable for tests. A lot take 15 seconds to timeout because of this.
+  // or CURATOR-720
   public static final int DEFAULT_ZK_CONNECT_TIMEOUT =
       Integer.getInteger("zkConnectTimeout", 15000);
 
diff --git a/solr/solrj/src/test/org/apache/solr/common/cloud/PerReplicaStatesIntegrationTest.java b/solr/solrj/src/test/org/apache/solr/common/cloud/PerReplicaStatesIntegrationTest.java
index 334851ca985..21a5307eb67 100644
--- a/solr/solrj/src/test/org/apache/solr/common/cloud/PerReplicaStatesIntegrationTest.java
+++ b/solr/solrj/src/test/org/apache/solr/common/cloud/PerReplicaStatesIntegrationTest.java
@@ -51,6 +51,7 @@
         + "org.apache.solr.common.cloud=INFO;"
         + "org.apache.solr.cloud.api.collections=INFO;"
         + "org.apache.solr.cloud.overseer=INFO")
+@SolrCloudTestCase.NoPrs
 public class PerReplicaStatesIntegrationTest extends SolrCloudTestCase {
 
   private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
diff --git a/solr/solrj/src/test/org/apache/solr/common/cloud/SolrZkClientCompressedDataTest.java b/solr/solrj/src/test/org/apache/solr/common/cloud/SolrZkClientCompressedDataTest.java
index 82682e5e122..dbdbbc39449 100644
--- a/solr/solrj/src/test/org/apache/solr/common/cloud/SolrZkClientCompressedDataTest.java
+++ b/solr/solrj/src/test/org/apache/solr/common/cloud/SolrZkClientCompressedDataTest.java
@@ -75,11 +75,8 @@ public void getData() throws Exception {
               + "\"leader\":\"true\"}}}}}}";
       byte[] arr = state.getBytes(StandardCharsets.UTF_8);
       byte[] compressedData = zLibStateCompression.compressBytes(arr);
-      ZkACLProvider aclProvider = new DefaultZkACLProvider();
       String path = ZkStateReader.COLLECTIONS_ZKNODE + "/c1/state.json";
-      zkClient
-          .getZooKeeper()
-          .create(path, compressedData, aclProvider.getACLsToAdd(path), CreateMode.PERSISTENT);
+      zkClient.create(path, compressedData, CreateMode.PERSISTENT, true);
 
       byte[] data =
           zkClient.getData(ZkStateReader.COLLECTIONS_ZKNODE + "/c1/state.json", null, null, true);
diff --git a/solr/test-framework/build.gradle b/solr/test-framework/build.gradle
index cb0ddabf7fa..2207ccbe515 100644
--- a/solr/test-framework/build.gradle
+++ b/solr/test-framework/build.gradle
@@ -43,6 +43,18 @@ dependencies {
   var zkExcludes = {
     exclude group: "org.apache.yetus", module: "audience-annotations"
   }
+  implementation('org.apache.curator:curator-client', {
+    exclude group: 'org.apache.zookeeper', module: 'zookeeper'
+  })
+  implementation('org.apache.curator:curator-framework', {
+    exclude group: 'org.apache.zookeeper', module: 'zookeeper'
+  })
+  // Necessary because ChaosMonkey uses KillSession from curator-test
+  implementation('org.apache.curator:curator-test') {
+    exclude group: 'org.apache.zookeeper', module: 'zookeeper'
+    exclude group: 'com.google.guava', module: 'guava'
+    exclude group: 'io.dropwizard.metrics', module: 'metrics-core'
+  }
 
   implementation('org.apache.zookeeper:zookeeper', zkExcludes)
   implementation('org.apache.zookeeper:zookeeper-jute', zkExcludes)
diff --git a/solr/test-framework/src/java/org/apache/solr/SolrIgnoredThreadsFilter.java b/solr/test-framework/src/java/org/apache/solr/SolrIgnoredThreadsFilter.java
index a3903e8ceaa..dc0e22dc6a3 100644
--- a/solr/test-framework/src/java/org/apache/solr/SolrIgnoredThreadsFilter.java
+++ b/solr/test-framework/src/java/org/apache/solr/SolrIgnoredThreadsFilter.java
@@ -82,6 +82,11 @@ public boolean reject(Thread t) {
       return true;
     }
 
+    // CURATOR-720
+    if (threadName.equals("OverseerExitThread")) {
+      return true;
+    }
+
     return threadName.startsWith("closeThreadPool");
   }
 }
diff --git a/solr/test-framework/src/java/org/apache/solr/SolrTestCaseJ4.java b/solr/test-framework/src/java/org/apache/solr/SolrTestCaseJ4.java
index 035fbcdef7f..27e9d7a9c33 100644
--- a/solr/test-framework/src/java/org/apache/solr/SolrTestCaseJ4.java
+++ b/solr/test-framework/src/java/org/apache/solr/SolrTestCaseJ4.java
@@ -94,7 +94,6 @@
 import org.apache.solr.client.solrj.util.ClientUtils;
 import org.apache.solr.cloud.IpTables;
 import org.apache.solr.cloud.MiniSolrCloudCluster;
-import org.apache.solr.cloud.TestConnectionStrategy;
 import org.apache.solr.common.SolrDocument;
 import org.apache.solr.common.SolrDocumentList;
 import org.apache.solr.common.SolrException;
@@ -286,7 +285,6 @@ public static void setupTestCases() {
 
     System.setProperty("zookeeper.forceSync", "no");
     System.setProperty("jetty.testMode", "true");
-    System.setProperty("solr.zookeeper.connectionStrategy", TestConnectionStrategy.class.getName());
     System.setProperty("enable.update.log", Boolean.toString(usually()));
     System.setProperty("tests.shardhandler.randomSeed", Long.toString(random().nextLong()));
     System.setProperty("solr.clustering.enabled", "false");
diff --git a/solr/test-framework/src/java/org/apache/solr/cloud/AbstractDigestZkACLAndCredentialsProvidersTestBase.java b/solr/test-framework/src/java/org/apache/solr/cloud/AbstractDigestZkACLAndCredentialsProvidersTestBase.java
index dbbf308bad1..31bfdb837c6 100644
--- a/solr/test-framework/src/java/org/apache/solr/cloud/AbstractDigestZkACLAndCredentialsProvidersTestBase.java
+++ b/solr/test-framework/src/java/org/apache/solr/cloud/AbstractDigestZkACLAndCredentialsProvidersTestBase.java
@@ -134,6 +134,8 @@ public void setUp() throws Exception {
     // no (or completely open) ACLs added. Therefore, hack your way into being authorized for
     // creating anyway
     zkClient
+        .getCuratorFramework()
+        .getZookeeperClient()
         .getZooKeeper()
         .addAuthInfo(
             "digest", (ALL_USERNAME + ":" + ALL_PASSWORD).getBytes(StandardCharsets.UTF_8));
@@ -305,6 +307,8 @@ public void testRepairACL() throws Exception {
       // no (or completely open) ACLs added. Therefore, hack your way into being authorized for
       // creating anyway
       zkClient
+          .getCuratorFramework()
+          .getZookeeperClient()
           .getZooKeeper()
           .addAuthInfo(
               "digest",
@@ -336,7 +340,7 @@ public void testRepairACL() throws Exception {
       NoAuthException e =
           assertThrows(
               NoAuthException.class, () -> zkClient.getData("/security.json", null, null, false));
-      assertEquals("/security.json", e.getPath());
+      assertEquals("/solr/security.json", e.getPath());
     }
   }
 
diff --git a/solr/test-framework/src/java/org/apache/solr/cloud/ChaosMonkey.java b/solr/test-framework/src/java/org/apache/solr/cloud/ChaosMonkey.java
index 01d07877a24..805866dc855 100644
--- a/solr/test-framework/src/java/org/apache/solr/cloud/ChaosMonkey.java
+++ b/solr/test-framework/src/java/org/apache/solr/cloud/ChaosMonkey.java
@@ -16,7 +16,6 @@
  */
 package org.apache.solr.cloud;
 
-import java.io.IOException;
 import java.lang.invoke.MethodHandles;
 import java.util.ArrayList;
 import java.util.List;
@@ -29,6 +28,7 @@
 import java.util.concurrent.TimeUnit;
 import java.util.concurrent.atomic.AtomicInteger;
 import java.util.regex.Pattern;
+import org.apache.curator.test.KillSession;
 import org.apache.lucene.tests.util.LuceneTestCase;
 import org.apache.solr.cloud.AbstractFullDistribZkTestBase.CloudJettyRunner;
 import org.apache.solr.common.cloud.DocCollection;
@@ -48,8 +48,6 @@
 import org.apache.solr.util.TestInjection;
 import org.apache.solr.util.TimeOut;
 import org.apache.zookeeper.KeeperException;
-import org.apache.zookeeper.TestableZooKeeper;
-import org.apache.zookeeper.ZooKeeper;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
@@ -151,8 +149,9 @@ public void expireSession(final JettySolrRunner jetty) {
     CoreContainer cores = jetty.getCoreContainer();
     if (cores != null) {
       monkeyLog("expire session for " + jetty.getLocalPort() + " !");
+      long sessionId = cores.getZkController().getZkClient().getZkSessionId();
+      zkServer.expire(sessionId);
       causeConnectionLoss(jetty);
-      zkServer.expire(cores.getZkController().getZkClient().getZooKeeper().getSessionId());
     }
   }
 
@@ -182,21 +181,11 @@ public static void causeConnectionLoss(JettySolrRunner jetty) {
     if (cores != null) {
       monkeyLog("Will cause connection loss on " + jetty.getLocalPort());
       SolrZkClient zkClient = cores.getZkController().getZkClient();
-      causeConnectionLoss(zkClient.getZooKeeper());
-    }
-  }
-
-  public static void causeConnectionLoss(ZooKeeper zooKeeper) {
-    assert zooKeeper instanceof TestableZooKeeper
-        : "Can only cause connection loss for TestableZookeeper";
-    if (zooKeeper instanceof TestableZooKeeper) {
       try {
-        ((TestableZooKeeper) zooKeeper).testableConnloss();
-      } catch (IOException ignored) {
-        // best effort
+        KillSession.kill(zkClient.getCuratorFramework().getZookeeperClient().getZooKeeper());
+      } catch (Exception e) {
+        log.warn("Exception causing connection loss", e);
       }
-    } else {
-      // TODO what now?
     }
   }
 
diff --git a/solr/test-framework/src/java/org/apache/solr/cloud/MiniSolrCloudCluster.java b/solr/test-framework/src/java/org/apache/solr/cloud/MiniSolrCloudCluster.java
index 26f33c7c60d..54f23d71260 100644
--- a/solr/test-framework/src/java/org/apache/solr/cloud/MiniSolrCloudCluster.java
+++ b/solr/test-framework/src/java/org/apache/solr/cloud/MiniSolrCloudCluster.java
@@ -54,6 +54,7 @@
 import java.util.concurrent.atomic.AtomicReference;
 import java.util.function.Consumer;
 import javax.servlet.Filter;
+import org.apache.curator.test.KillSession;
 import org.apache.lucene.tests.util.LuceneTestCase;
 import org.apache.solr.SolrTestCaseJ4;
 import org.apache.solr.client.solrj.embedded.SSLConfig;
@@ -788,7 +789,14 @@ public void expireZkSession(JettySolrRunner jetty) {
     CoreContainer cores = jetty.getCoreContainer();
     if (cores != null) {
       ChaosMonkey.causeConnectionLoss(jetty);
-      zkServer.expire(cores.getZkController().getZkClient().getZooKeeper().getSessionId());
+      SolrZkClient zkClient = cores.getZkController().getZkClient();
+      long sessionId = zkClient.getZkSessionId();
+      zkServer.expire(sessionId);
+      try {
+        KillSession.kill(zkClient.getCuratorFramework().getZookeeperClient().getZooKeeper());
+      } catch (Exception e) {
+        log.error("Exception killing session", e);
+      }
       if (log.isInfoEnabled()) {
         log.info("Expired zookeeper session from node {}", jetty.getBaseUrl());
       }
diff --git a/solr/test-framework/src/java/org/apache/solr/cloud/ZkTestServer.java b/solr/test-framework/src/java/org/apache/solr/cloud/ZkTestServer.java
index 14d2a5a1f86..8bd7396c7a7 100644
--- a/solr/test-framework/src/java/org/apache/solr/cloud/ZkTestServer.java
+++ b/solr/test-framework/src/java/org/apache/solr/cloud/ZkTestServer.java
@@ -45,7 +45,6 @@
 import org.apache.solr.common.util.Utils;
 import org.apache.zookeeper.CreateMode;
 import org.apache.zookeeper.KeeperException;
-import org.apache.zookeeper.Op;
 import org.apache.zookeeper.WatchedEvent;
 import org.apache.zookeeper.Watcher;
 import org.apache.zookeeper.ZooDefs;
@@ -796,42 +795,26 @@ public void buildZooKeeper(Path solrhome, String config, String schema) throws E
     props.put("configName", "conf1");
     final ZkNodeProps zkProps = new ZkNodeProps(props);
 
-    List<Op> ops = new ArrayList<>(2);
-    String path = "/collections";
-    ops.add(
-        Op.create(
-            path, null, chRootClient.getZkACLProvider().getACLsToAdd(path), CreateMode.PERSISTENT));
-    path = "/collections/collection1";
-    ops.add(
-        Op.create(
-            path,
-            Utils.toJSON(zkProps),
-            chRootClient.getZkACLProvider().getACLsToAdd(path),
-            CreateMode.PERSISTENT));
-    path = "/collections/collection1/shards";
-    ops.add(
-        Op.create(
-            path, null, chRootClient.getZkACLProvider().getACLsToAdd(path), CreateMode.PERSISTENT));
-    path = "/collections/control_collection";
-    ops.add(
-        Op.create(
-            path,
-            Utils.toJSON(zkProps),
-            chRootClient.getZkACLProvider().getACLsToAdd(path),
-            CreateMode.PERSISTENT));
-    path = "/collections/control_collection/shards";
-    ops.add(
-        Op.create(
-            path, null, chRootClient.getZkACLProvider().getACLsToAdd(path), CreateMode.PERSISTENT));
-    path = "/configs";
-    ops.add(
-        Op.create(
-            path, null, chRootClient.getZkACLProvider().getACLsToAdd(path), CreateMode.PERSISTENT));
-    path = "/configs/conf1";
-    ops.add(
-        Op.create(
-            path, null, chRootClient.getZkACLProvider().getACLsToAdd(path), CreateMode.PERSISTENT));
-    chRootClient.multi(ops, true);
+    chRootClient.multi(
+        op -> op.create().withMode(CreateMode.PERSISTENT).forPath("/collections", null),
+        op ->
+            op.create()
+                .withMode(CreateMode.PERSISTENT)
+                .forPath("/collections/collection1", Utils.toJSON(zkProps)),
+        op ->
+            op.create()
+                .withMode(CreateMode.PERSISTENT)
+                .forPath("/collections/collection1/shards", null),
+        op ->
+            op.create()
+                .withMode(CreateMode.PERSISTENT)
+                .forPath("/collections/control_collection", Utils.toJSON(zkProps)),
+        op ->
+            op.create()
+                .withMode(CreateMode.PERSISTENT)
+                .forPath("/collections/control_collection/shards", null),
+        op -> op.create().withMode(CreateMode.PERSISTENT).forPath("/configs", null),
+        op -> op.create().withMode(CreateMode.PERSISTENT).forPath("/configs/conf1", null));
 
     // for now, always upload the config and schema to the canonical names
     putConfig("conf1", chRootClient, solrhome, config, "solrconfig.xml");
diff --git a/versions.lock b/versions.lock
index f1d2e911ede..7418e07bfd9 100644
--- a/versions.lock
+++ b/versions.lock
@@ -48,7 +48,7 @@ com.google.cloud:google-cloud-storage:2.40.1 (2 constraints: cf1cc626)
 com.google.code.gson:gson:2.11.0 (6 constraints: 0c550bc0)
 com.google.errorprone:error_prone_annotations:2.28.0 (15 constraints: a5c51259)
 com.google.guava:failureaccess:1.0.2 (2 constraints: fb19bf37)
-com.google.guava:guava:33.1.0-jre (26 constraints: 0280374a)
+com.google.guava:guava:33.1.0-jre (27 constraints: 698fe64d)
 com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava (2 constraints: 4b35b0a0)
 com.google.http-client:google-http-client:1.44.2 (11 constraints: 45bf29b8)
 com.google.http-client:google-http-client-apache-v2:1.44.2 (2 constraints: b9201d76)
@@ -100,7 +100,7 @@ edu.ucar:netcdf4:4.5.5 (1 constraints: 650c0402)
 edu.ucar:udunits:4.5.5 (1 constraints: 2b06034e)
 edu.usc.ir:sentiment-analysis-parser:0.1 (1 constraints: fa0b50e9)
 io.dropwizard.metrics:metrics-annotation:4.2.26 (1 constraints: 361073b0)
-io.dropwizard.metrics:metrics-core:4.2.26 (9 constraints: e382b9e2)
+io.dropwizard.metrics:metrics-core:4.2.26 (10 constraints: b190d0d8)
 io.dropwizard.metrics:metrics-graphite:4.2.26 (1 constraints: 4005473b)
 io.dropwizard.metrics:metrics-healthchecks:4.2.26 (1 constraints: 1d1183c9)
 io.dropwizard.metrics:metrics-jetty10:4.2.26 (1 constraints: 4005473b)
@@ -194,9 +194,10 @@ org.apache.commons:commons-exec:1.4.0 (2 constraints: 031132cf)
 org.apache.commons:commons-lang3:3.15.0 (6 constraints: cc4e807b)
 org.apache.commons:commons-math3:3.6.1 (5 constraints: 57322799)
 org.apache.commons:commons-text:1.12.0 (2 constraints: 651f97e5)
-org.apache.curator:curator-client:5.7.0 (2 constraints: ec14cea3)
-org.apache.curator:curator-framework:5.7.0 (2 constraints: 0914ad75)
-org.apache.curator:curator-recipes:5.7.0 (1 constraints: 0e051936)
+org.apache.curator:curator-client:5.7.1 (2 constraints: ee14fda3)
+org.apache.curator:curator-framework:5.7.1 (2 constraints: 0b14da75)
+org.apache.curator:curator-recipes:5.7.1 (1 constraints: 0f051a36)
+org.apache.curator:curator-test:5.7.1 (1 constraints: 0f051a36)
 org.apache.hadoop:hadoop-annotations:3.4.0 (1 constraints: 09050636)
 org.apache.hadoop:hadoop-auth:3.4.0 (1 constraints: 09050636)
 org.apache.hadoop:hadoop-client-api:3.4.0 (3 constraints: 16281b5e)
@@ -283,9 +284,9 @@ org.apache.tika:tika-core:1.28.5 (2 constraints: d8118f11)
 org.apache.tika:tika-parsers:1.28.5 (1 constraints: 42054a3b)
 org.apache.tomcat:annotations-api:6.0.53 (1 constraints: 40054e3b)
 org.apache.xmlbeans:xmlbeans:5.0.3 (2 constraints: 72173075)
-org.apache.zookeeper:zookeeper:3.9.2 (3 constraints: a51e5e0a)
+org.apache.zookeeper:zookeeper:3.9.2 (2 constraints: 15105ed4)
 org.apache.zookeeper:zookeeper-jute:3.9.2 (2 constraints: 9d12b123)
-org.apiguardian:apiguardian-api:1.1.2 (2 constraints: 601bd5a8)
+org.apiguardian:apiguardian-api:1.1.2 (4 constraints: b03c1624)
 org.bitbucket.b_c:jose4j:0.9.6 (2 constraints: 14106bd4)
 org.bouncycastle:bcmail-jdk15on:1.70 (1 constraints: 310c8af5)
 org.bouncycastle:bcpkix-jdk15on:1.70 (2 constraints: ce1b11b3)
@@ -354,6 +355,9 @@ org.jetbrains.kotlin:kotlin-stdlib:1.9.10 (4 constraints: 5c405537)
 org.jetbrains.kotlin:kotlin-stdlib-common:1.9.10 (3 constraints: 3a2c8a72)
 org.jetbrains.kotlin:kotlin-stdlib-jdk7:1.9.10 (1 constraints: e210ffd2)
 org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.9.10 (4 constraints: e23809e7)
+org.junit:junit-bom:5.6.2 (2 constraints: 67212a70)
+org.junit.jupiter:junit-jupiter-api:5.6.2 (2 constraints: e516c64d)
+org.junit.platform:junit-platform-commons:1.6.2 (2 constraints: 7c1841a1)
 org.locationtech.jts:jts-core:1.19.0 (2 constraints: a31de760)
 org.locationtech.jts.io:jts-io-common:1.19.0 (1 constraints: 930d513a)
 org.locationtech.proj4j:proj4j:1.2.2 (1 constraints: 5d0daf2c)
@@ -361,6 +365,7 @@ org.locationtech.spatial4j:spatial4j:0.8 (1 constraints: 59105498)
 org.lz4:lz4-java:1.8.0 (1 constraints: fc0cd91b)
 org.opengis:geoapi:3.0.1 (7 constraints: 1361d046)
 org.openjdk.jmh:jmh-core:1.37 (1 constraints: df04fc30)
+org.opentest4j:opentest4j:1.2.0 (1 constraints: 670f0e7d)
 org.osgi:org.osgi.resource:1.0.0 (1 constraints: e60f2999)
 org.osgi:org.osgi.service.serviceloader:1.0.0 (1 constraints: e60f2999)
 org.osgi:osgi.annotation:8.1.0 (1 constraints: 0b051636)
@@ -386,7 +391,7 @@ org.tallison:metadata-extractor:2.17.1.0 (1 constraints: f00c3b28)
 org.tallison.xmp:xmpcore-shaded:6.1.10 (1 constraints: 300e8d49)
 org.threeten:threetenbp:1.6.9 (4 constraints: 2833ea68)
 org.tukaani:xz:1.9 (1 constraints: 030c5be9)
-org.xerial.snappy:snappy-java:1.1.10.5 (5 constraints: be46ae36)
+org.xerial.snappy:snappy-java:1.1.10.5 (6 constraints: 17552375)
 software.amazon.awssdk:annotations:2.26.19 (28 constraints: 48a79471)
 software.amazon.awssdk:apache-client:2.26.19 (4 constraints: 112adae3)
 software.amazon.awssdk:arns:2.26.19 (2 constraints: 231878c1)
diff --git a/versions.props b/versions.props
index 1c681b6ebfc..0433da0fc4f 100644
--- a/versions.props
+++ b/versions.props
@@ -39,7 +39,7 @@ org.apache.commons:commons-configuration2=2.11.0
 org.apache.commons:commons-exec=1.4.0
 org.apache.commons:commons-lang3=3.15.0
 org.apache.commons:commons-math3=3.6.1
-org.apache.curator:*=5.7.0
+org.apache.curator:*=5.7.1
 org.apache.hadoop.thirdparty:*=1.2.0
 org.apache.hadoop:*=3.4.0
 org.apache.httpcomponents:httpclient=4.5.14

From 1e69bc00dcbb23232fe35f25606e00c7850c92cd Mon Sep 17 00:00:00 2001
From: Sanjay Dutt <sanjaydutt@apache.org>
Date: Thu, 31 Oct 2024 21:03:48 +0530
Subject: [PATCH 13/32] Remove crave.yaml (#2821)

---
 crave.yaml | 2 --
 1 file changed, 2 deletions(-)
 delete mode 100644 crave.yaml

diff --git a/crave.yaml b/crave.yaml
deleted file mode 100644
index 91a7980e70f..00000000000
--- a/crave.yaml
+++ /dev/null
@@ -1,2 +0,0 @@
-Solr:
-  image: "accupara/openjdk:21"
\ No newline at end of file

From cc30093c5ee988555389b50cf2333edf743bb50f Mon Sep 17 00:00:00 2001
From: Sanjay Dutt <sanjaydutt@apache.org>
Date: Thu, 31 Oct 2024 21:39:48 +0530
Subject: [PATCH 14/32] SOLR-17448 SOLR-17497: IndexFetcher, catch exception
 instead of bubbling up uncaught (#2800)

---
 solr/core/src/java/org/apache/solr/handler/IndexFetcher.java | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/solr/core/src/java/org/apache/solr/handler/IndexFetcher.java b/solr/core/src/java/org/apache/solr/handler/IndexFetcher.java
index 9aa35440d97..8cce5ed129f 100644
--- a/solr/core/src/java/org/apache/solr/handler/IndexFetcher.java
+++ b/solr/core/src/java/org/apache/solr/handler/IndexFetcher.java
@@ -88,6 +88,7 @@
 import org.apache.lucene.index.IndexCommit;
 import org.apache.lucene.index.IndexWriter;
 import org.apache.lucene.index.SegmentInfos;
+import org.apache.lucene.store.AlreadyClosedException;
 import org.apache.lucene.store.Directory;
 import org.apache.lucene.store.FSDirectory;
 import org.apache.lucene.store.FilterDirectory;
@@ -1800,7 +1801,7 @@ private void fetch() throws Exception {
             () -> {
               try {
                 file.sync();
-              } catch (IOException e) {
+              } catch (IOException | AlreadyClosedException e) {
                 fsyncException = e;
               }
             });

From 8f7f617e6a2150eb16648792184eb1304c50ce0d Mon Sep 17 00:00:00 2001
From: Eric Pugh <epugh@opensourceconnections.com>
Date: Fri, 1 Nov 2024 08:02:32 -0400
Subject: [PATCH 15/32] SOLR-17524: Call out a few changes in 10. (#2808)

We can add more as we get closer.

Co-authored-by: Christos Malliaridis <c.malliaridis@gmail.com>
---
 .../upgrade-notes/pages/major-changes-in-solr-10.adoc    | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-10.adoc b/solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-10.adoc
index 516141d317c..af10bc83cff 100644
--- a/solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-10.adoc
+++ b/solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-10.adoc
@@ -35,6 +35,15 @@ Minimum Java version for Solr 10.x is Java 21.
 The Solr CLI has gone through some significant renovations to reduce technical debt, and now functions more consistently and predictably. Most notably, CLI commands now use unix-style options, e.g. `--help` instead of `-help`.
 Users are urged to review all use of the `bin/solr` command, such as in scripts and documentation, to ensure that the correct options are being used.
 
+Some key changes that you may run into are:
+
+ * Commands that interact with Solr now all use `--solr-url` (or `-s`) plus a `--name` (or `-c`) to specify the Solr to interact with.
+ * You may use `--zk-host` (or `-z`) in place of the `--solr-url` option to lookup the Solr host.
+ * Basic Authentication support has been added to bin/solr via the `--credentials` (or `-u`) option.
+ * Some short and single-letter options have been removed to avoid conflicts or in favor to other options.
+ 
+To learn about the updated options in each CLI tool, use the `--help` option or look up the tool in the documentation.
+
 === SolrJ
 
 * Starting in 10, the Maven POM for SolrJ does not refer to SolrJ modules like ZooKeeper.  If you require such functionality, you need to add additional dependencies.

From d96c6859821dadcd11187030684c00e6e8975480 Mon Sep 17 00:00:00 2001
From: Eric Pugh <epugh@opensourceconnections.com>
Date: Fri, 1 Nov 2024 13:03:11 -0400
Subject: [PATCH 16/32] SOLR-17352: Remove deprecated options from CLI (#2820)

Options deprecated in 9x line of Solr are removed from the Solr CLI.
---
 solr/CHANGES.txt                              |   2 +
 solr/bin/solr                                 |  47 ++---
 solr/bin/solr.cmd                             |  45 ++---
 solr/bin/solr.in.cmd                          |   4 +-
 .../src/java/org/apache/solr/cli/ApiTool.java |  31 +---
 .../java/org/apache/solr/cli/AssertTool.java  | 162 +-----------------
 .../java/org/apache/solr/cli/AuthTool.java    |  61 -------
 .../java/org/apache/solr/cli/ClusterTool.java |   3 +-
 .../solr/cli/ConfigSetDownloadTool.java       |  48 +-----
 .../apache/solr/cli/ConfigSetUploadTool.java  |  48 +-----
 .../java/org/apache/solr/cli/ConfigTool.java  |  41 +----
 .../java/org/apache/solr/cli/CreateTool.java  |  73 +-------
 .../java/org/apache/solr/cli/DeleteTool.java  |  63 +------
 .../java/org/apache/solr/cli/ExportTool.java  |  33 +---
 .../org/apache/solr/cli/HealthcheckTool.java  |   3 -
 .../org/apache/solr/cli/LinkConfigTool.java   |   3 +-
 .../java/org/apache/solr/cli/PackageTool.java |  21 +--
 .../org/apache/solr/cli/PostLogsTool.java     |  22 +--
 .../java/org/apache/solr/cli/PostTool.java    |  99 +----------
 .../org/apache/solr/cli/RunExampleTool.java   |  18 +-
 .../apache/solr/cli/SnapshotCreateTool.java   |   1 -
 .../apache/solr/cli/SnapshotDeleteTool.java   |   1 -
 .../apache/solr/cli/SnapshotDescribeTool.java |   1 -
 .../apache/solr/cli/SnapshotExportTool.java   |  18 --
 .../org/apache/solr/cli/SnapshotListTool.java |   1 -
 .../src/java/org/apache/solr/cli/SolrCLI.java |  96 ++---------
 .../java/org/apache/solr/cli/StatusTool.java  |   2 +-
 .../org/apache/solr/cli/UpdateACLTool.java    |   2 +-
 .../java/org/apache/solr/cli/ZkCpTool.java    |   6 +-
 .../java/org/apache/solr/cli/ZkLsTool.java    |   6 +-
 .../org/apache/solr/cli/ZkMkrootTool.java     |   3 -
 .../java/org/apache/solr/cli/ZkMvTool.java    |   8 +-
 .../java/org/apache/solr/cli/ZkRmTool.java    |   6 +-
 .../org/apache/solr/cli/CreateToolTest.java   |   2 +-
 .../org/apache/solr/cli/DeleteToolTest.java   |   4 +-
 .../org/apache/solr/cli/PackageToolTest.java  |   2 +-
 solr/packaging/test/test_bats.bats            |   2 +-
 solr/packaging/test/test_config.bats          |  18 +-
 solr/packaging/test/test_post.bats            |  32 ++--
 solr/packaging/test/test_postlogs.bats        |   2 +-
 solr/packaging/test/test_prometheus.bats      |   4 +-
 solr/packaging/test/test_snapshots.bats       |   9 +-
 solr/packaging/test/test_ssl.bats             |   3 +-
 solr/packaging/test/test_version.bats         |  18 +-
 solr/packaging/test/test_zk.bats              |  18 +-
 solr/prometheus-exporter/bin/solr-exporter    |   8 +-
 .../prometheus-exporter/bin/solr-exporter.cmd |   8 +-
 .../prometheus/exporter/SolrExporter.java     | 140 ++-------------
 .../deployment-guide/pages/enabling-ssl.adoc  |   6 +-
 ...onitoring-with-prometheus-and-grafana.adoc |  16 +-
 50 files changed, 178 insertions(+), 1092 deletions(-)

diff --git a/solr/CHANGES.txt b/solr/CHANGES.txt
index 83327d8605b..8ed458efdb0 100644
--- a/solr/CHANGES.txt
+++ b/solr/CHANGES.txt
@@ -78,6 +78,8 @@ Deprecation Removals
 
 * SOLR-17494: Remove language specific writer types (i.e wt= ruby, python, php, and phps). (Eric Pugh)
 
+* SOLR-17352: Remove deprecated Solr CLI options.  Run bin/solr yourcommand -h to see current options.  (Eric Pugh, Christos Malliardis)
+
 Dependency Upgrades
 ---------------------
 (No changes)
diff --git a/solr/bin/solr b/solr/bin/solr
index d036a696400..f51011444e1 100755
--- a/solr/bin/solr
+++ b/solr/bin/solr
@@ -369,7 +369,7 @@ function print_usage() {
     echo ""
     echo "Usage: solr $CMD [-f] [--user-managed] [--host host] [-p port] [--server-dir directory] [-z zkHost] [-m memory] [-e example] [--solr-home solr.solr.home] [--data-home solr.data.home] [--jvm-opts \"jvm-opts\"] [--verbose]"
     echo ""
-    echo "  -f                    Start Solr in foreground; default starts Solr in the background"
+    echo "  -f/--foreground       Start Solr in foreground; default starts Solr in the background"
     echo "                          and sends stdout / stderr to solr-PORT-console.log"
     echo ""
     echo "  --user-managed        Start Solr in user managed aka standalone mode"
@@ -403,7 +403,7 @@ function print_usage() {
     echo "  --data-home <dir>     Sets the solr.data.home system property, where Solr will store index data in <instance_dir>/data subdirectories."
     echo "                          If not set, Solr uses solr.solr.home for config and data."
     echo ""
-    echo "  -e <example>          Name of the example to run; available examples:"
+    echo "  -e/--example <name>   Name of the example to run; available examples:"
     echo "      cloud:              SolrCloud example"
     echo "      techproducts:       Comprehensive example illustrating many of Solr's core capabilities"
     echo "      schemaless:         Schema-less example (schema is inferred from data during indexing)"
@@ -419,7 +419,7 @@ function print_usage() {
     echo "                          you could pass: -j \"--include-jetty-dir=/etc/jetty/custom/server/\""
     echo "                          In most cases, you should wrap the additional parameters in double quotes."
     echo ""
-    echo "  --no-prompt            Don't prompt for input; accept all defaults when running examples that accept user input"
+    echo "  -y/--no-prompt        Don't prompt for input; accept all defaults when running examples that accept user input"
     echo ""
     echo "  --force               If attempting to start Solr as the root user, the script will exit with a warning that running Solr as \"root\" can cause problems."
     echo "                          It is possible to override this warning with the '--force' parameter."
@@ -574,7 +574,7 @@ function stop_solr() {
 
 if [ $# -eq 1 ]; then
   case $1 in
-    --help|-h|-help)
+    --help|-h)
         run_tool ""
         exit
     ;;
@@ -694,7 +694,7 @@ if [ $# -gt 0 ]; then
             PASS_TO_RUN_EXAMPLE+=("--user-managed")
             shift
         ;;
-        -d|--dir|-dir|--server-dir)
+        --server-dir)
             if [[ -z "$2" || "${2:0:1}" == "-" ]]; then
               print_usage "$SCRIPT_CMD" "Server directory is required when using the $1 option!"
               exit 1
@@ -714,7 +714,7 @@ if [ $# -gt 0 ]; then
             SOLR_SERVER_DIR="$(cd "$SOLR_SERVER_DIR" || (echo "SOLR_SERVER_DIR not found" && exit 1); pwd)"
             shift 2
         ;;
-        -s|--solr-home|-solr.home)
+        --solr-home)
             if [[ -z "$2" || "${2:0:1}" == "-" ]]; then
               print_usage "$SCRIPT_CMD" "Solr home directory is required when using the $1 option!"
               exit 1
@@ -723,11 +723,15 @@ if [ $# -gt 0 ]; then
             SOLR_HOME="$2"
             shift 2
         ;;
-        -t|--data-home|-data.home)
+        --data-home)
+            if [[ -z "$2" || "${2:0:1}" == "-" ]]; then
+              print_usage "$SCRIPT_CMD" "Data home directory is required when using the $1 option!"
+              exit 1
+            fi        
             SOLR_DATA_HOME="$2"
             shift 2
         ;;
-        -e|--example|-example)
+        -e|--example)
             if [[ -z "$2" || "${2:0:1}" == "-" ]]; then
               print_usage "$SCRIPT_CMD" "Example name is required when using the $1 option!"
               exit 1
@@ -735,11 +739,11 @@ if [ $# -gt 0 ]; then
             EXAMPLE="$2"
             shift 2
         ;;
-        -f|--foreground|-foreground)
+        -f|--foreground)
             FG="true"
             shift
         ;;
-        --host|-host)
+        --host)
             if [[ -z "$2" || "${2:0:1}" == "-" ]]; then
               print_usage "$SCRIPT_CMD" "Hostname is required when using the $1 option!"
               exit 1
@@ -748,7 +752,7 @@ if [ $# -gt 0 ]; then
             PASS_TO_RUN_EXAMPLE+=("--host" "$SOLR_HOST")
             shift 2
         ;;
-        -m|--memory|-memory)
+        -m|--memory)
             if [[ -z "$2" || "${2:0:1}" == "-" ]]; then
               print_usage "$SCRIPT_CMD" "Memory setting is required when using the $1 option!"
               exit 1
@@ -757,7 +761,7 @@ if [ $# -gt 0 ]; then
             PASS_TO_RUN_EXAMPLE+=("-m" "$SOLR_HEAP")
             shift 2
         ;;
-        -p|--port|-port)
+        -p|--port)
             if [[ -z "$2" || "${2:0:1}" == "-" ]]; then
               print_usage "$SCRIPT_CMD" "Port number is required when using the $1 option!"
               exit 1
@@ -767,7 +771,7 @@ if [ $# -gt 0 ]; then
             PASS_TO_RUN_EXAMPLE+=("-p" "$SOLR_PORT")
             shift 2
         ;;
-        -z|--zk-host|-zkHost|--zkHost)
+        -z|--zk-host)
             if [[ -z "$2" || "${2:0:1}" == "-" ]]; then
               print_usage "$SCRIPT_CMD" "Zookeeper connection string is required when using the $1 option!"
               exit 1
@@ -776,7 +780,7 @@ if [ $# -gt 0 ]; then
             PASS_TO_RUN_EXAMPLE+=("-z" "$ZK_HOST")
             shift 2
         ;;
-        -a|--jvm-opts|-addlopts)
+        --jvm-opts)
             if [[ -z "$2" ]]; then
               print_usage "$SCRIPT_CMD" "JVM options are required when using the $1 option!"
               exit 1
@@ -785,16 +789,15 @@ if [ $# -gt 0 ]; then
             PASS_TO_RUN_EXAMPLE+=("--jvm-opts" "$ADDITIONAL_CMD_OPTS")
             shift 2
         ;;
-        -j|--jettyconfig|-jettyconfig)
+        -j|--jettyconfig)
             if [[ -z "$2" || "${2:0:1}" == "-" ]]; then
               print_usage "$SCRIPT_CMD" "Jetty config is required when using the $1 option!"
               exit 1
             fi
             ADDITIONAL_JETTY_CONFIG="$2"
-            PASS_TO_RUN_EXAMPLE+=("-j" "$ADDITIONAL_JETTY_CONFIG")
             shift 2
         ;;
-        -k|--key|-key)
+        -k|--key)
             if [[ -z "$2" || "${2:0:1}" == "-" ]]; then
               print_usage "$SCRIPT_CMD" "Stop key is required when using the $1 option!"
               exit 1
@@ -802,15 +805,15 @@ if [ $# -gt 0 ]; then
             STOP_KEY="$2"
             shift 2
         ;;
-        -h|--help|-help)
+        -h|--help)
             print_usage "$SCRIPT_CMD"
             exit 0
         ;;
-        --noprompt|-noprompt|--no-prompt)
+        -y|--no-prompt)
             PASS_TO_RUN_EXAMPLE+=("--no-prompt")
             shift
         ;;
-        -V|--verbose|-verbose|-v)
+        --verbose)
             verbose=true
             SOLR_LOG_LEVEL=DEBUG
             PASS_TO_RUN_EXAMPLE+=("--verbose")
@@ -820,11 +823,11 @@ if [ $# -gt 0 ]; then
             SOLR_LOG_LEVEL=WARN
             shift
         ;;
-        --all|-all)
+        --all)
             stop_all=true
             shift
         ;;
-        --force|-force)
+        --force)
             FORCE=true
             PASS_TO_RUN_EXAMPLE+=("--force")
             shift
diff --git a/solr/bin/solr.cmd b/solr/bin/solr.cmd
index ce6f2aea386..783a4feed00 100755
--- a/solr/bin/solr.cmd
+++ b/solr/bin/solr.cmd
@@ -248,17 +248,12 @@ set FIRST_ARG=%1
 
 IF [%1]==[] goto usage
 
-REM -help is a special case to faciliate folks learning about how to use Solr.
-IF "%1"=="-help" goto run_solrcli
-IF "%1"=="-usage" goto run_solrcli
 IF "%1"=="-h" goto run_solrcli
 IF "%1"=="--help" goto run_solrcli
-IF "%1"=="-help" goto run_solrcli
-IF "%1"=="/?" goto run_solrcli
 IF "%1"=="status" goto run_solrcli
 IF "%1"=="version" goto run_solrcli
 IF "%1"=="-v" goto run_solrcli
-IF "%1"=="-version" goto run_solrcli
+IF "%1"=="--version" goto run_solrcli
 IF "%1"=="assert" goto run_solrcli
 IF "%1"=="zk" goto run_solrcli
 IF "%1"=="export" goto run_solrcli
@@ -287,11 +282,8 @@ goto parse_args
 :usage
 IF NOT "%SCRIPT_ERROR%"=="" ECHO %SCRIPT_ERROR%
 IF [%FIRST_ARG%]==[] goto run_solrcli
-IF "%FIRST_ARG%"=="-help" goto run_solrcli
-IF "%FIRST_ARG%"=="-usage" goto run_solrcli
 IF "%FIRST_ARG%"=="-h" goto run_solrcli
 IF "%FIRST_ARG%"=="--help" goto run_solrcli
-IF "%FIRST_ARG%"=="/?" goto run_solrcli
 IF "%SCRIPT_CMD%"=="start" goto start_usage
 IF "%SCRIPT_CMD%"=="restart" goto start_usage
 IF "%SCRIPT_CMD%"=="stop" goto stop_usage
@@ -310,7 +302,7 @@ goto done
 @echo.
 @echo Usage: solr %SCRIPT_CMD% [-f] [--user-managed] [--host hostname] [-p port] [--server-dir directory] [-z zkHost] [-m memory] [-e example] [--solr-home solr.solr.home] [--data-home solr.data.home] [--jvm-opts "jvm-opts"] [--verbose]
 @echo.
-@echo   -f            Start Solr in foreground; default starts Solr in the background
+@echo   -f/--foreground Start Solr in foreground; default starts Solr in the background
 @echo                   and sends stdout / stderr to solr-PORT-console.log
 @echo.
 @echo   --user-managed Start Solr in user managed aka standalone mode"
@@ -318,7 +310,7 @@ goto done
 @echo.
 @echo   --host host   Specify the hostname for this Solr instance
 @echo.
-@echo   -p port       Specify the port to start the Solr HTTP listener on; default is 8983
+@echo   -p/--port port Specify the port to start the Solr HTTP listener on; default is 8983
 @echo                   The specified port (SOLR_PORT) will also be used to determine the stop port
 @echo                   STOP_PORT=(%%SOLR_PORT%%-1000) and JMX RMI listen port RMI_PORT=(%%SOLR_PORT%%+10000).
 @echo                   For instance, if you set -p 8985, then the STOP_PORT=7985 and RMI_PORT=18985
@@ -330,10 +322,10 @@ goto done
 @echo                   an embedded ZooKeeper instance will be launched.
 @echo                   Set the ZK_CREATE_CHROOT environment variable to true if your ZK host has a chroot path, and you want to create it automatically."
 @echo.
-@echo   -m memory     Sets the min (-Xms) and max (-Xmx) heap size for the JVM, such as: -m 4g
+@echo   -m/--memory memory Sets the min (-Xms) and max (-Xmx) heap size for the JVM, such as: -m 4g
 @echo                   results in: -Xms4g -Xmx4g; by default, this script sets the heap size to 512m
 @echo.
-@echo   --solr.home dir  Sets the solr.solr.home system property; Solr will create core directories under
+@echo   --solr-home dir  Sets the solr.solr.home system property; Solr will create core directories under
 @echo                   this directory. This allows you to run multiple Solr instances on the same host
 @echo                   while reusing the same server directory set using the --server-dir parameter. If set, the
 @echo                   specified directory should contain a solr.xml file, unless solr.xml exists in Zookeeper.
@@ -344,7 +336,7 @@ goto done
 @echo   --data-home dir Sets the solr.data.home system property, where Solr will store index data in ^<instance_dir^>/data subdirectories.
 @echo                   If not set, Solr uses solr.solr.home for both config and data.
 @echo.
-@echo   -e example    Name of the example to run; available examples:
+@echo   -e/--example name Name of the example to run; available examples:
 @echo       cloud:          SolrCloud example
 @echo       techproducts:   Comprehensive example illustrating many of Solr's core capabilities
 @echo       schemaless:     Schema-less example (schema is inferred from data during indexing)
@@ -360,7 +352,7 @@ goto done
 @echo                 you could pass: -j "--include-jetty-dir=/etc/jetty/custom/server/"
 @echo                 In most cases, you should wrap the additional parameters in double quotes.
 @echo.
-@echo   --no-prompt   Don't prompt for input; accept all defaults when running examples that accept user input
+@echo   -y/--no-prompt Don't prompt for input; accept all defaults when running examples that accept user input
 @echo.
 @echo   --verbose and -q/--quiet Verbose or quiet logging. Sets default log level to DEBUG or WARN instead of INFO
 @echo.
@@ -390,25 +382,16 @@ set "arg=%~1"
 set "firstTwo=%arg:~0,2%"
 IF "%SCRIPT_CMD%"=="" set SCRIPT_CMD=start
 IF [%1]==[] goto process_script_cmd
-IF "%1"=="-help" goto usage
+IF "%1"=="--help" goto usage
 IF "%1"=="-h" goto usage
-IF "%1"=="-usage" goto usage
-IF "%1"=="/?" goto usage
 IF "%1"=="-f" goto set_foreground_mode
 IF "%1"=="--foreground" goto set_foreground_mode
-IF "%1"=="-V" goto set_verbose
 IF "%1"=="--verbose" goto set_verbose
-IF "%1"=="-v" goto set_verbose
 IF "%1"=="-q" goto set_warn
 IF "%1"=="--quiet" goto set_warn
 IF "%1"=="--user-managed" goto set_user_managed_mode
-IF "%1"=="-d" goto set_server_dir
-IF "%1"=="--dir" goto set_server_dir
 IF "%1"=="--server-dir" goto set_server_dir
-IF "%1"=="-s" goto set_solr_home_dir
 IF "%1"=="--solr-home" goto set_solr_home_dir
-IF "%1"=="-t" goto set_solr_data_dir
-IF "%1"=="--solr-data" goto set_solr_data_dir
 IF "%1"=="--data-home" goto set_solr_data_dir
 IF "%1"=="-e" goto set_example
 IF "%1"=="--example" goto set_example
@@ -419,21 +402,16 @@ IF "%1"=="-p" goto set_port
 IF "%1"=="--port" goto set_port
 IF "%1"=="-z" goto set_zookeeper
 IF "%1"=="--zk-host" goto set_zookeeper
-IF "%1"=="-zkHost" goto set_zookeeper
-IF "%1"=="--zkHost" goto set_zookeeper
 IF "%1"=="-s" goto set_solr_url
 IF "%1"=="--solr-url" goto set_solr_url
-IF "%1"=="-solrUrl" goto set_solr_url
-IF "%1"=="-a" goto set_jvm_opts
 IF "%1"=="--jvm-opts" goto set_jvm_opts
 IF "%1"=="-j" goto set_addl_jetty_config
 IF "%1"=="--jettyconfig" goto set_addl_jetty_config
-IF "%1"=="--noprompt" goto set_noprompt
 IF "%1"=="--no-prompt" goto set_noprompt
+IF "%1"=="-y" goto set_noprompt
 IF "%1"=="-k" goto set_stop_key
 IF "%1"=="--key" goto set_stop_key
 IF "%1"=="--all" goto set_stop_all
-IF "%1"=="-all" goto set_stop_all
 IF "%firstTwo%"=="-D" goto set_passthru
 IF NOT "%1"=="" goto invalid_cmd_line
 goto invalid_cmd_line
@@ -1180,10 +1158,7 @@ goto done
 IF [%1]==[] goto run_config
 IF "%1"=="-z" goto set_config_zk
 IF "%1"=="--zk-host" goto set_config_zk
-IF "%1"=="-zkHost" goto set_config_zk
-IF "%1"=="--zkHost" goto set_config_zk
-IF "%1"=="-s" goto set_config_url_scheme
-IF "%1"=="-scheme" goto set_config_url_scheme
+IF "%1"=="--scheme" goto set_config_url_scheme
 set "CONFIG_ARGS=!CONFIG_ARGS! %1"
 SHIFT
 goto parse_config_args
diff --git a/solr/bin/solr.in.cmd b/solr/bin/solr.in.cmd
index 9c2d9822eb2..ac058eae131 100755
--- a/solr/bin/solr.in.cmd
+++ b/solr/bin/solr.in.cmd
@@ -90,7 +90,7 @@ REM set RMI_PORT=18983
 
 REM Anything you add to the SOLR_OPTS variable will be included in the java
 REM start command line as-is, in ADDITION to other options. If you specify the
-REM -a option on start script, those options will be appended as well. Examples:
+REM --jvm-opts option on start script, those options will be appended as well. Examples:
 REM set SOLR_OPTS=%SOLR_OPTS% -Dsolr.autoSoftCommit.maxTime=3000
 REM set SOLR_OPTS=%SOLR_OPTS% -Dsolr.autoCommit.maxTime=60000
 
@@ -255,4 +255,4 @@ REM set SOLR_MODULES=extraction,ltr
 
 REM Configure the default replica placement plugin to use if one is not configured in cluster properties
 REM See https://solr.apache.org/guide/solr/latest/configuration-guide/replica-placement-plugins.html for details
-REM set SOLR_PLACEMENTPLUGIN_DEFAULT=simple
\ No newline at end of file
+REM set SOLR_PLACEMENTPLUGIN_DEFAULT=simple
diff --git a/solr/core/src/java/org/apache/solr/cli/ApiTool.java b/solr/core/src/java/org/apache/solr/cli/ApiTool.java
index 76c6aa6f60f..e9af6116fa9 100644
--- a/solr/core/src/java/org/apache/solr/cli/ApiTool.java
+++ b/solr/core/src/java/org/apache/solr/cli/ApiTool.java
@@ -21,7 +21,6 @@
 import java.net.URI;
 import java.util.List;
 import org.apache.commons.cli.CommandLine;
-import org.apache.commons.cli.DeprecatedAttributes;
 import org.apache.commons.cli.Option;
 import org.apache.solr.client.solrj.SolrRequest;
 import org.apache.solr.client.solrj.impl.JsonMapResponseParser;
@@ -57,20 +56,7 @@ public List<Option> getOptions() {
             .longOpt("solr-url")
             .argName("URL")
             .hasArg()
-            .required(false) // swap back to required when we eliminate deprecated option
-            .desc("Send a GET request to a Solr API endpoint.")
-            .build(),
-        Option.builder("get")
-            .longOpt("get")
-            .deprecated(
-                DeprecatedAttributes.builder()
-                    .setForRemoval(true)
-                    .setSince("9.7")
-                    .setDescription("Use --solr-url instead")
-                    .get())
-            .argName("URL")
-            .hasArg()
-            .required(false)
+            .required(true)
             .desc("Send a GET request to a Solr API endpoint.")
             .build(),
         SolrCLI.OPTION_CREDENTIALS);
@@ -78,16 +64,11 @@ public List<Option> getOptions() {
 
   @Override
   public void runImpl(CommandLine cli) throws Exception {
-    String response = null;
-    String getUrl =
-        cli.hasOption("solr-url") ? cli.getOptionValue("solr-url") : cli.getOptionValue("get");
-    if (getUrl != null) {
-      response = callGet(getUrl, cli.getOptionValue(SolrCLI.OPTION_CREDENTIALS.getLongOpt()));
-    }
-    if (response != null) {
-      // pretty-print the response to stdout
-      echo(response);
-    }
+    String getUrl = cli.getOptionValue("solr-url");
+    String response = callGet(getUrl, cli.getOptionValue(SolrCLI.OPTION_CREDENTIALS.getLongOpt()));
+
+    // pretty-print the response to stdout
+    echo(response);
   }
 
   protected String callGet(String url, String credentials) throws Exception {
diff --git a/solr/core/src/java/org/apache/solr/cli/AssertTool.java b/solr/core/src/java/org/apache/solr/cli/AssertTool.java
index 5f5d27c3ee0..966e7993146 100644
--- a/solr/core/src/java/org/apache/solr/cli/AssertTool.java
+++ b/solr/core/src/java/org/apache/solr/cli/AssertTool.java
@@ -26,7 +26,6 @@
 import java.util.List;
 import java.util.concurrent.TimeUnit;
 import org.apache.commons.cli.CommandLine;
-import org.apache.commons.cli.DeprecatedAttributes;
 import org.apache.commons.cli.Option;
 import org.apache.solr.client.solrj.SolrClient;
 import org.apache.solr.client.solrj.SolrServerException;
@@ -61,54 +60,14 @@ public String getName() {
   @Override
   public List<Option> getOptions() {
     return List.of(
-        Option.builder("R")
-            .desc("Asserts that we are NOT the root user.")
-            .deprecated(
-                DeprecatedAttributes.builder()
-                    .setForRemoval(true)
-                    .setSince("9.8")
-                    .setDescription("Use --not-root instead")
-                    .get())
-            .build(),
         Option.builder().desc("Asserts that we are NOT the root user.").longOpt("not-root").build(),
-        Option.builder("r")
-            .desc("Asserts that we are the root user.")
-            .deprecated(
-                DeprecatedAttributes.builder()
-                    .setForRemoval(true)
-                    .setSince("9.8")
-                    .setDescription("Use --root instead")
-                    .get())
-            .build(),
         Option.builder().desc("Asserts that we are the root user.").longOpt("root").build(),
-        Option.builder("S")
-            .desc("Asserts that Solr is NOT running on a certain URL. Default timeout is 1000ms.")
-            .deprecated(
-                DeprecatedAttributes.builder()
-                    .setForRemoval(true)
-                    .setSince("9.8")
-                    .setDescription("Use --not-started instead")
-                    .get())
-            .hasArg(true)
-            .argName("url")
-            .build(),
         Option.builder()
             .desc("Asserts that Solr is NOT running on a certain URL. Default timeout is 1000ms.")
             .longOpt("not-started")
             .hasArg(true)
             .argName("url")
             .build(),
-        Option.builder("s")
-            .desc("Asserts that Solr is running on a certain URL. Default timeout is 1000ms.")
-            .deprecated(
-                DeprecatedAttributes.builder()
-                    .setForRemoval(true)
-                    .setSince("9.8")
-                    .setDescription("Use --started instead")
-                    .get())
-            .hasArg(true)
-            .argName("url")
-            .build(),
         Option.builder()
             .desc("Asserts that Solr is running on a certain URL. Default timeout is 1000ms.")
             .longOpt("started")
@@ -121,52 +80,18 @@ public List<Option> getOptions() {
             .hasArg(true)
             .argName("directory")
             .build(),
-        Option.builder("x")
-            .desc("Asserts that directory <directory> exists.")
-            .deprecated(
-                DeprecatedAttributes.builder()
-                    .setForRemoval(true)
-                    .setSince("9.8")
-                    .setDescription("Use --exists instead")
-                    .get())
-            .hasArg(true)
-            .argName("directory")
-            .build(),
         Option.builder()
             .desc("Asserts that directory <directory> exists.")
             .longOpt("exists")
             .hasArg(true)
             .argName("directory")
             .build(),
-        Option.builder("X")
-            .desc("Asserts that directory <directory> does NOT exist.")
-            .deprecated(
-                DeprecatedAttributes.builder()
-                    .setForRemoval(true)
-                    .setSince("9.8")
-                    .setDescription("Use --not-exists instead")
-                    .get())
-            .hasArg(true)
-            .argName("directory")
-            .build(),
         Option.builder()
             .desc("Asserts that directory <directory> does NOT exist.")
             .longOpt("not-exists")
             .hasArg(true)
             .argName("directory")
             .build(),
-        Option.builder("c")
-            .desc(
-                "Asserts that Solr is running in cloud mode.  Also fails if Solr not running.  URL should be for root Solr path.")
-            .deprecated(
-                DeprecatedAttributes.builder()
-                    .setForRemoval(true)
-                    .setSince("9.8")
-                    .setDescription("Use --cloud instead")
-                    .get())
-            .hasArg(true)
-            .argName("url")
-            .build(),
         Option.builder()
             .desc(
                 "Asserts that Solr is running in cloud mode.  Also fails if Solr not running.  URL should be for root Solr path.")
@@ -174,18 +99,6 @@ public List<Option> getOptions() {
             .hasArg(true)
             .argName("url")
             .build(),
-        Option.builder("C")
-            .desc(
-                "Asserts that Solr is not running in cloud mode.  Also fails if Solr not running.  URL should be for root Solr path.")
-            .deprecated(
-                DeprecatedAttributes.builder()
-                    .setForRemoval(true)
-                    .setSince("9.8")
-                    .setDescription("Use --not-cloud instead")
-                    .get())
-            .hasArg(true)
-            .argName("url")
-            .build(),
         Option.builder()
             .desc(
                 "Asserts that Solr is not running in cloud mode.  Also fails if Solr not running.  URL should be for root Solr path.")
@@ -193,35 +106,12 @@ public List<Option> getOptions() {
             .hasArg(true)
             .argName("url")
             .build(),
-        Option.builder("m")
-            .desc("Exception message to be used in place of the default error message.")
-            .deprecated(
-                DeprecatedAttributes.builder()
-                    .setForRemoval(true)
-                    .setSince("9.8")
-                    .setDescription("Use --message instead")
-                    .get())
-            .hasArg(true)
-            .argName("message")
-            .build(),
         Option.builder()
             .desc("Exception message to be used in place of the default error message.")
             .longOpt("message")
             .hasArg(true)
             .argName("message")
             .build(),
-        Option.builder("t")
-            .desc("Timeout in ms for commands supporting a timeout.")
-            .deprecated(
-                DeprecatedAttributes.builder()
-                    .setForRemoval(true)
-                    .setSince("9.8")
-                    .setDescription("Use --timeout instead")
-                    .get())
-            .hasArg(true)
-            .type(Long.class)
-            .argName("ms")
-            .build(),
         Option.builder()
             .desc("Timeout in ms for commands supporting a timeout.")
             .longOpt("timeout")
@@ -229,15 +119,6 @@ public List<Option> getOptions() {
             .type(Long.class)
             .argName("ms")
             .build(),
-        Option.builder("e")
-            .desc("Return an exit code instead of printing error message on assert fail.")
-            .deprecated(
-                DeprecatedAttributes.builder()
-                    .setForRemoval(true)
-                    .setSince("9.8")
-                    .setDescription("Use --exitcode instead")
-                    .get())
-            .build(),
         Option.builder()
             .desc("Return an exit code instead of printing error message on assert fail.")
             .longOpt("exitcode")
@@ -250,7 +131,7 @@ public List<Option> getOptions() {
    * Otherwise, very similar to the parent runTool method.
    *
    * @param cli the command line object
-   * @return 0 on success, or a number corresponding to number of tests that failed, or 100 for a
+   * @return 0 on success, or a number corresponding to number of tests that failed, or 100 for an
    *     Error
    * @throws Exception if a tool failed, e.g. authentication failure
    */
@@ -297,82 +178,47 @@ protected int runAssert(CommandLine cli) throws Exception {
     if (cli.hasOption("message")) {
       message = cli.getOptionValue("message");
     }
-    if (cli.hasOption("t")) {
-      timeoutMs = Long.parseLong(cli.getOptionValue("t"));
-    }
     if (cli.hasOption("timeout")) {
-      timeoutMs = Long.parseLong(cli.getOptionValue("timeout"));
-    }
-    if (cli.hasOption("e")) {
-      useExitCode = true;
+      timeoutMs = cli.getParsedOptionValue("timeout");
     }
     if (cli.hasOption("exitcode")) {
       useExitCode = true;
     }
 
     int ret = 0;
-    if (cli.hasOption("root") || cli.hasOption("r")) {
+    if (cli.hasOption("root")) {
       ret += assertRootUser();
     }
-    if (cli.hasOption("not-root") || cli.hasOption("R")) {
+    if (cli.hasOption("not-root")) {
       ret += assertNotRootUser();
     }
-    if (cli.hasOption("x")) {
-      ret += assertFileExists(cli.getOptionValue("x"));
-    }
     if (cli.hasOption("exists")) {
       ret += assertFileExists(cli.getOptionValue("exists"));
     }
-    if (cli.hasOption("X")) {
-      ret += assertFileNotExists(cli.getOptionValue("X"));
-    }
     if (cli.hasOption("not-exists")) {
       ret += assertFileNotExists(cli.getOptionValue("not-exists"));
     }
     if (cli.hasOption("same-user")) {
       ret += sameUser(cli.getOptionValue("same-user"));
     }
-    if (cli.hasOption("s")) {
-      ret +=
-          assertSolrRunning(
-              cli.getOptionValue("s"), cli.getOptionValue(SolrCLI.OPTION_CREDENTIALS.getLongOpt()));
-    }
     if (cli.hasOption("started")) {
       ret +=
           assertSolrRunning(
               cli.getOptionValue("started"),
               cli.getOptionValue(SolrCLI.OPTION_CREDENTIALS.getLongOpt()));
     }
-    if (cli.hasOption("S")) {
-      ret +=
-          assertSolrNotRunning(
-              SolrCLI.getOptionWithDeprecatedAndDefault(cli, "not-started", "S", null),
-              cli.getOptionValue(SolrCLI.OPTION_CREDENTIALS.getLongOpt()));
-    }
     if (cli.hasOption("not-started")) {
       ret +=
           assertSolrNotRunning(
               cli.getOptionValue("not-started"),
               cli.getOptionValue(SolrCLI.OPTION_CREDENTIALS.getLongOpt()));
     }
-    if (cli.hasOption("c")) {
-      ret +=
-          assertSolrRunningInCloudMode(
-              SolrCLI.normalizeSolrUrl(cli.getOptionValue("c")),
-              cli.getOptionValue(SolrCLI.OPTION_CREDENTIALS.getLongOpt()));
-    }
     if (cli.hasOption("cloud")) {
       ret +=
           assertSolrRunningInCloudMode(
               SolrCLI.normalizeSolrUrl(cli.getOptionValue("cloud")),
               cli.getOptionValue(SolrCLI.OPTION_CREDENTIALS.getLongOpt()));
     }
-    if (cli.hasOption("C")) {
-      ret +=
-          assertSolrNotRunningInCloudMode(
-              SolrCLI.normalizeSolrUrl(cli.getOptionValue("C")),
-              cli.getOptionValue(SolrCLI.OPTION_CREDENTIALS.getLongOpt()));
-    }
     if (cli.hasOption("not-cloud")) {
       ret +=
           assertSolrNotRunningInCloudMode(
diff --git a/solr/core/src/java/org/apache/solr/cli/AuthTool.java b/solr/core/src/java/org/apache/solr/cli/AuthTool.java
index 6000e1e8667..bc091073170 100644
--- a/solr/core/src/java/org/apache/solr/cli/AuthTool.java
+++ b/solr/core/src/java/org/apache/solr/cli/AuthTool.java
@@ -35,7 +35,6 @@
 import java.util.Locale;
 import java.util.stream.Collectors;
 import org.apache.commons.cli.CommandLine;
-import org.apache.commons.cli.DeprecatedAttributes;
 import org.apache.commons.cli.HelpFormatter;
 import org.apache.commons.cli.Option;
 import org.apache.lucene.util.Constants;
@@ -112,20 +111,6 @@ public List<Option> getOptions() {
             .hasArg()
             .argName("true|false")
             .build(),
-        Option.builder("blockUnknown")
-            .longOpt("blockUnknown")
-            .deprecated(
-                DeprecatedAttributes.builder()
-                    .setForRemoval(true)
-                    .setSince("9.8")
-                    .setDescription("Use --block-unknown instead")
-                    .get())
-            .hasArg()
-            .argName("true|false")
-            .required(false)
-            .desc(
-                "Blocks all access for unknown users (requires authentication for all endpoints).")
-            .build(),
         Option.builder()
             .longOpt("solr-include-file")
             .hasArg()
@@ -133,20 +118,6 @@ public List<Option> getOptions() {
             .desc(
                 "The Solr include file which contains overridable environment variables for configuring Solr configurations.")
             .build(),
-        Option.builder("solrIncludeFile")
-            .longOpt("solrIncludeFile")
-            .deprecated(
-                DeprecatedAttributes.builder()
-                    .setForRemoval(true)
-                    .setSince("9.7")
-                    .setDescription("Use --solr-include-file instead")
-                    .get())
-            .hasArg()
-            .argName("FILE")
-            .required(false)
-            .desc(
-                "The Solr include file which contains overridable environment variables for configuring Solr configurations.")
-            .build(),
         Option.builder()
             .longOpt("update-include-file-only")
             .desc(
@@ -154,21 +125,6 @@ public List<Option> getOptions() {
                     + " authentication (i.e. don't update security.json).")
             .hasArg()
             .build(),
-        Option.builder()
-            .longOpt("updateIncludeFileOnly")
-            .deprecated(
-                DeprecatedAttributes.builder()
-                    .setForRemoval(true)
-                    .setSince("9.8")
-                    .setDescription("Use --update-include-file-only instead")
-                    .get())
-            .hasArg()
-            .argName("true|false")
-            .required(false)
-            .desc(
-                "Only update the solr.in.sh or solr.in.cmd file, and skip actual enabling/disabling"
-                    + " authentication (i.e. don't update security.json).")
-            .build(),
         Option.builder()
             .longOpt("auth-conf-dir")
             .hasArg()
@@ -177,25 +133,8 @@ public List<Option> getOptions() {
             .desc(
                 "This is where any authentication related configuration files, if any, would be placed.")
             .build(),
-        Option.builder("solrIncludeFile")
-            .longOpt("solrIncludeFile")
-            .deprecated(
-                DeprecatedAttributes.builder()
-                    .setForRemoval(true)
-                    .setSince("9.7")
-                    .setDescription("Use --solr-include-file instead")
-                    .get())
-            .hasArg()
-            .argName("FILE")
-            .required(false)
-            .desc(
-                "The Solr include file which contains overridable environment variables for configuring Solr configurations.")
-            .build(),
         SolrCLI.OPTION_SOLRURL,
-        SolrCLI.OPTION_SOLRURL_DEPRECATED,
-        SolrCLI.OPTION_SOLRURL_DEPRECATED_SHORT,
         SolrCLI.OPTION_ZKHOST,
-        SolrCLI.OPTION_ZKHOST_DEPRECATED,
         SolrCLI.OPTION_CREDENTIALS);
   }
 
diff --git a/solr/core/src/java/org/apache/solr/cli/ClusterTool.java b/solr/core/src/java/org/apache/solr/cli/ClusterTool.java
index 23300bc8800..8b6bd17f56b 100644
--- a/solr/core/src/java/org/apache/solr/cli/ClusterTool.java
+++ b/solr/core/src/java/org/apache/solr/cli/ClusterTool.java
@@ -66,8 +66,7 @@ public List<Option> getOptions() {
             .required(false)
             .desc("Set the property to this value.")
             .build(),
-        SolrCLI.OPTION_ZKHOST,
-        SolrCLI.OPTION_ZKHOST_DEPRECATED);
+        SolrCLI.OPTION_ZKHOST);
   }
 
   @Override
diff --git a/solr/core/src/java/org/apache/solr/cli/ConfigSetDownloadTool.java b/solr/core/src/java/org/apache/solr/cli/ConfigSetDownloadTool.java
index 2a987b7bdce..b0e5aaa6d4a 100644
--- a/solr/core/src/java/org/apache/solr/cli/ConfigSetDownloadTool.java
+++ b/solr/core/src/java/org/apache/solr/cli/ConfigSetDownloadTool.java
@@ -23,7 +23,6 @@
 import java.nio.file.Paths;
 import java.util.List;
 import org.apache.commons.cli.CommandLine;
-import org.apache.commons.cli.DeprecatedAttributes;
 import org.apache.commons.cli.Option;
 import org.apache.solr.common.cloud.SolrZkClient;
 import org.slf4j.Logger;
@@ -48,47 +47,18 @@ public List<Option> getOptions() {
             .longOpt("conf-name")
             .hasArg()
             .argName("NAME")
-            .required(false) // should be true, but we have deprecated option as well.
-            .desc("Configset name in ZooKeeper.")
-            .build(),
-        Option.builder()
-            .longOpt("confname")
-            .hasArg()
-            .argName("NAME")
-            .deprecated(
-                DeprecatedAttributes.builder()
-                    .setForRemoval(true)
-                    .setSince("9.8")
-                    .setDescription("Use --conf-name instead")
-                    .get())
-            .required(false)
+            .required(true)
             .desc("Configset name in ZooKeeper.")
             .build(),
         Option.builder("d")
             .longOpt("conf-dir")
             .hasArg()
             .argName("DIR")
-            .required(false) // should be true, but we have deprecated option as well.
-            .desc("Local directory with configs.")
-            .build(),
-        Option.builder()
-            .longOpt("confdir")
-            .hasArg()
-            .argName("NAME")
-            .deprecated(
-                DeprecatedAttributes.builder()
-                    .setForRemoval(true)
-                    .setSince("9.8")
-                    .setDescription("Use --conf-dir instead")
-                    .get())
-            .required(false)
+            .required(true)
             .desc("Local directory with configs.")
             .build(),
         SolrCLI.OPTION_SOLRURL,
-        SolrCLI.OPTION_SOLRURL_DEPRECATED,
-        SolrCLI.OPTION_SOLRURL_DEPRECATED_SHORT,
         SolrCLI.OPTION_ZKHOST,
-        SolrCLI.OPTION_ZKHOST_DEPRECATED,
         SolrCLI.OPTION_CREDENTIALS);
   }
 
@@ -107,17 +77,11 @@ public void runImpl(CommandLine cli) throws Exception {
     SolrCLI.raiseLogLevelUnlessVerbose(cli);
     String zkHost = SolrCLI.getZkHost(cli);
 
-    try (SolrZkClient zkClient = SolrCLI.getSolrZkClient(cli, zkHost)) {
-      echoIfVerbose("\nConnecting to ZooKeeper at " + zkHost + " ...");
-      String confName =
-          cli.hasOption("conf-name")
-              ? cli.getOptionValue("conf-name")
-              : cli.getOptionValue("confname");
-      String confDir =
-          cli.hasOption("conf-dir")
-              ? cli.getOptionValue("conf-dir")
-              : cli.getOptionValue("confdir");
+    String confName = cli.getOptionValue("conf-name");
+    String confDir = cli.getOptionValue("conf-dir");
 
+    echoIfVerbose("\nConnecting to ZooKeeper at " + zkHost + " ...");
+    try (SolrZkClient zkClient = SolrCLI.getSolrZkClient(cli, zkHost)) {
       Path configSetPath = Paths.get(confDir);
       // we try to be nice about having the "conf" in the directory, and we create it if it's not
       // there.
diff --git a/solr/core/src/java/org/apache/solr/cli/ConfigSetUploadTool.java b/solr/core/src/java/org/apache/solr/cli/ConfigSetUploadTool.java
index 3799b498c4e..597b901c7d0 100644
--- a/solr/core/src/java/org/apache/solr/cli/ConfigSetUploadTool.java
+++ b/solr/core/src/java/org/apache/solr/cli/ConfigSetUploadTool.java
@@ -22,7 +22,6 @@
 import java.nio.file.Paths;
 import java.util.List;
 import org.apache.commons.cli.CommandLine;
-import org.apache.commons.cli.DeprecatedAttributes;
 import org.apache.commons.cli.Option;
 import org.apache.solr.common.cloud.SolrZkClient;
 import org.apache.solr.common.cloud.ZkMaintenanceUtils;
@@ -50,47 +49,18 @@ public List<Option> getOptions() {
             .longOpt("conf-name")
             .hasArg()
             .argName("NAME")
-            .required(false) // should be true, but we have deprecated option as well.
-            .desc("Configset name in ZooKeeper.")
-            .build(),
-        Option.builder()
-            .longOpt("confname")
-            .hasArg()
-            .argName("NAME")
-            .deprecated(
-                DeprecatedAttributes.builder()
-                    .setForRemoval(true)
-                    .setSince("9.8")
-                    .setDescription("Use --conf-name instead")
-                    .get())
-            .required(false)
+            .required(true)
             .desc("Configset name in ZooKeeper.")
             .build(),
         Option.builder("d")
             .longOpt("conf-dir")
             .hasArg()
             .argName("DIR")
-            .required(false) // should be true, but we have deprecated option as well.
-            .desc("Local directory with configs.")
-            .build(),
-        Option.builder()
-            .longOpt("confdir")
-            .hasArg()
-            .argName("DIR")
-            .deprecated(
-                DeprecatedAttributes.builder()
-                    .setForRemoval(true)
-                    .setSince("9.8")
-                    .setDescription("Use --conf-dir instead")
-                    .get())
-            .required(false)
+            .required(true)
             .desc("Local directory with configs.")
             .build(),
         SolrCLI.OPTION_SOLRURL,
-        SolrCLI.OPTION_SOLRURL_DEPRECATED,
-        SolrCLI.OPTION_SOLRURL_DEPRECATED_SHORT,
         SolrCLI.OPTION_ZKHOST,
-        SolrCLI.OPTION_ZKHOST_DEPRECATED,
         SolrCLI.OPTION_CREDENTIALS);
   }
 
@@ -112,15 +82,11 @@ public void runImpl(CommandLine cli) throws Exception {
     final String solrInstallDir = System.getProperty("solr.install.dir");
     Path solrInstallDirPath = Paths.get(solrInstallDir);
 
-    String confName =
-        cli.hasOption("conf-name")
-            ? cli.getOptionValue("conf-name")
-            : cli.getOptionValue("confname");
-    String confDir =
-        cli.hasOption("conf-dir") ? cli.getOptionValue("conf-dir") : cli.getOptionValue("confdir");
-    try (SolrZkClient zkClient = SolrCLI.getSolrZkClient(cli, zkHost)) {
-      echoIfVerbose("\nConnecting to ZooKeeper at " + zkHost + " ...");
+    String confName = cli.getOptionValue("conf-name");
+    String confDir = cli.getOptionValue("conf-dir");
 
+    echoIfVerbose("\nConnecting to ZooKeeper at " + zkHost + " ...");
+    try (SolrZkClient zkClient = SolrCLI.getSolrZkClient(cli, zkHost)) {
       final Path configsetsDirPath = SolrCLI.getConfigSetsDir(solrInstallDirPath);
       Path confPath = ConfigSetService.getConfigsetPath(confDir, configsetsDirPath.toString());
 
@@ -128,7 +94,7 @@ public void runImpl(CommandLine cli) throws Exception {
           "Uploading "
               + confPath.toAbsolutePath()
               + " for config "
-              + cli.getOptionValue("conf-name")
+              + confName
               + " to ZooKeeper at "
               + zkHost);
       FileTypeMagicUtil.assertConfigSetFolderLegal(confPath);
diff --git a/solr/core/src/java/org/apache/solr/cli/ConfigTool.java b/solr/core/src/java/org/apache/solr/cli/ConfigTool.java
index f1a90c071a7..5cc3f0909d6 100644
--- a/solr/core/src/java/org/apache/solr/cli/ConfigTool.java
+++ b/solr/core/src/java/org/apache/solr/cli/ConfigTool.java
@@ -22,7 +22,6 @@
 import java.util.List;
 import java.util.Map;
 import org.apache.commons.cli.CommandLine;
-import org.apache.commons.cli.DeprecatedAttributes;
 import org.apache.commons.cli.MissingArgumentException;
 import org.apache.commons.cli.Option;
 import org.apache.solr.client.solrj.SolrClient;
@@ -72,37 +71,10 @@ public List<Option> getOptions() {
             .longOpt("property")
             .argName("PROP")
             .hasArg()
-            .required(
-                false) // Should be TRUE but have a deprecated option to deal with first, so we
-            // enforce in code
-            .desc(
-                "Name of the Config API property to apply the action to, such as: 'updateHandler.autoSoftCommit.maxTime'.")
-            .build(),
-        Option.builder("p")
-            .deprecated(
-                DeprecatedAttributes.builder()
-                    .setForRemoval(true)
-                    .setSince("9.8")
-                    .setDescription("Use --property instead")
-                    .get())
-            .hasArg()
-            .argName("PROP")
-            .required(false)
+            .required(true)
             .desc(
                 "Name of the Config API property to apply the action to, such as: 'updateHandler.autoSoftCommit.maxTime'.")
             .build(),
-        Option.builder("v")
-            .deprecated(
-                DeprecatedAttributes.builder()
-                    .setForRemoval(true)
-                    .setSince("9.8")
-                    .setDescription("Use --value instead")
-                    .get())
-            .argName("VALUE")
-            .hasArg()
-            .required(false)
-            .desc("Set the property to this value; accepts JSON objects and strings.")
-            .build(),
         Option.builder()
             .longOpt("value")
             .argName("VALUE")
@@ -111,10 +83,7 @@ public List<Option> getOptions() {
             .desc("Set the property to this value; accepts JSON objects and strings.")
             .build(),
         SolrCLI.OPTION_SOLRURL,
-        SolrCLI.OPTION_SOLRURL_DEPRECATED,
-        SolrCLI.OPTION_SOLRURL_DEPRECATED_SHORT,
         SolrCLI.OPTION_ZKHOST,
-        SolrCLI.OPTION_ZKHOST_DEPRECATED,
         SolrCLI.OPTION_CREDENTIALS);
   }
 
@@ -123,12 +92,8 @@ public void runImpl(CommandLine cli) throws Exception {
     String solrUrl = SolrCLI.normalizeSolrUrl(cli);
     String action = cli.getOptionValue("action", "set-property");
     String collection = cli.getOptionValue("name");
-    String property = SolrCLI.getOptionWithDeprecatedAndDefault(cli, "property", "p", null);
-    String value = SolrCLI.getOptionWithDeprecatedAndDefault(cli, "value", "v", null);
-
-    if (property == null) {
-      throw new MissingArgumentException("'property' is a required option.");
-    }
+    String property = cli.getOptionValue("property");
+    String value = cli.getOptionValue("value");
 
     // value is required unless the property is one of the unset- type.
     if (!action.contains("unset-") && value == null) {
diff --git a/solr/core/src/java/org/apache/solr/cli/CreateTool.java b/solr/core/src/java/org/apache/solr/cli/CreateTool.java
index 41742c600a8..6c14ed6e21e 100644
--- a/solr/core/src/java/org/apache/solr/cli/CreateTool.java
+++ b/solr/core/src/java/org/apache/solr/cli/CreateTool.java
@@ -27,7 +27,6 @@
 import java.util.Set;
 import java.util.concurrent.TimeUnit;
 import org.apache.commons.cli.CommandLine;
-import org.apache.commons.cli.DeprecatedAttributes;
 import org.apache.commons.cli.Option;
 import org.apache.commons.io.FileUtils;
 import org.apache.commons.io.file.PathUtils;
@@ -97,20 +96,6 @@ public List<Option> getOptions() {
             .desc(
                 "Number of copies of each document across the collection (replicas per shard); default is 1.")
             .build(),
-        Option.builder()
-            .longOpt("replicationFactor")
-            .deprecated(
-                DeprecatedAttributes.builder()
-                    .setForRemoval(true)
-                    .setSince("9.7")
-                    .setDescription("Use --replication-factor instead")
-                    .get())
-            .hasArg()
-            .argName("#")
-            .required(false)
-            .desc(
-                "Number of copies of each document across the collection (replicas per shard); default is 1.")
-            .build(),
         Option.builder("d")
             .longOpt("conf-dir")
             .argName("DIR")
@@ -120,22 +105,6 @@ public List<Option> getOptions() {
                     + SolrCLI.DEFAULT_CONFIG_SET
                     + '.')
             .build(),
-        Option.builder("confdir")
-            .longOpt("confdir")
-            .deprecated(
-                DeprecatedAttributes.builder()
-                    .setForRemoval(true)
-                    .setSince("9.7")
-                    .setDescription("Use --conf-dir instead")
-                    .get())
-            .argName("DIR")
-            .hasArg()
-            .required(false)
-            .desc(
-                "Configuration directory to copy when creating the new collection; default is "
-                    + SolrCLI.DEFAULT_CONFIG_SET
-                    + '.')
-            .build(),
         Option.builder("n")
             .longOpt("conf-name")
             .argName("NAME")
@@ -143,24 +112,8 @@ public List<Option> getOptions() {
             .required(false)
             .desc("Configuration name; default is the collection name.")
             .build(),
-        Option.builder("confname")
-            .longOpt("confname")
-            .deprecated(
-                DeprecatedAttributes.builder()
-                    .setForRemoval(true)
-                    .setSince("9.7")
-                    .setDescription("Use --conf-name instead")
-                    .get())
-            .argName("NAME")
-            .hasArg()
-            .required(false)
-            .desc("Configuration name; default is the collection name.")
-            .build(),
         SolrCLI.OPTION_SOLRURL,
-        SolrCLI.OPTION_SOLRURL_DEPRECATED,
-        SolrCLI.OPTION_SOLRURL_DEPRECATED_SHORT,
         SolrCLI.OPTION_ZKHOST,
-        SolrCLI.OPTION_ZKHOST_DEPRECATED,
         SolrCLI.OPTION_CREDENTIALS);
   }
 
@@ -182,10 +135,7 @@ protected void createCore(CommandLine cli, SolrClient solrClient) throws Excepti
     String solrUrl = cli.getOptionValue("solr-url", SolrCLI.getDefaultSolrUrl());
 
     final String solrInstallDir = System.getProperty("solr.install.dir");
-    final String confDirName =
-        cli.hasOption("confdir")
-            ? cli.getOptionValue("confdir")
-            : cli.getOptionValue("conf-dir", SolrCLI.DEFAULT_CONFIG_SET);
+    final String confDirName = cli.getOptionValue("conf-dir", SolrCLI.DEFAULT_CONFIG_SET);
 
     // we allow them to pass a directory instead of a configset name
     Path configsetDir = Paths.get(confDirName);
@@ -268,14 +218,8 @@ protected void createCollection(CloudSolrClient cloudSolrClient, CommandLine cli
 
     String collectionName = cli.getOptionValue("name");
     final String solrInstallDir = System.getProperty("solr.install.dir");
-    String confName =
-        cli.hasOption("conf-name")
-            ? cli.getOptionValue("conf-name")
-            : cli.getOptionValue("confname");
-    String confDir =
-        cli.hasOption("confdir")
-            ? cli.getOptionValue("confdir")
-            : cli.getOptionValue("conf-dir", SolrCLI.DEFAULT_CONFIG_SET);
+    String confName = cli.getOptionValue("conf-name");
+    String confDir = cli.getOptionValue("conf-dir", SolrCLI.DEFAULT_CONFIG_SET);
     Path solrInstallDirPath = Paths.get(solrInstallDir);
     Path confDirPath = Paths.get(confDir);
     ensureConfDirExists(solrInstallDirPath, confDirPath);
@@ -299,8 +243,6 @@ protected void createCollection(CloudSolrClient cloudSolrClient, CommandLine cli
 
     if (cli.hasOption("replication-factor")) {
       replicationFactor = Integer.parseInt(cli.getOptionValue("replication-factor"));
-    } else if (cli.hasOption("replicationFactor")) {
-      replicationFactor = Integer.parseInt(cli.getOptionValue("replicationFactor"));
     }
 
     boolean configExistsInZk =
@@ -397,16 +339,13 @@ private void ensureConfDirExists(Path solrInstallDir, Path confDirName) {
   }
 
   private void printDefaultConfigsetWarningIfNecessary(CommandLine cli) {
-    final String confDirectoryName =
-        cli.hasOption("confdir")
-            ? cli.getOptionValue("confdir")
-            : cli.getOptionValue("conf-dir", SolrCLI.DEFAULT_CONFIG_SET);
-    final String confName = cli.getOptionValue("confname", "");
+    final String confDirectoryName = cli.getOptionValue("conf-dir", SolrCLI.DEFAULT_CONFIG_SET);
+    final String confName = cli.getOptionValue("conf-name", "");
 
     if (confDirectoryName.equals("_default")
         && (confName.equals("") || confName.equals("_default"))) {
       final String collectionName = cli.getOptionValue("name");
-      final String solrUrl = cli.getOptionValue("solrUrl", SolrCLI.getDefaultSolrUrl());
+      final String solrUrl = cli.getOptionValue("solr-url", SolrCLI.getDefaultSolrUrl());
       final String curlCommand =
           String.format(
               Locale.ROOT,
diff --git a/solr/core/src/java/org/apache/solr/cli/DeleteTool.java b/solr/core/src/java/org/apache/solr/cli/DeleteTool.java
index c89a3fb0301..069128663b4 100644
--- a/solr/core/src/java/org/apache/solr/cli/DeleteTool.java
+++ b/solr/core/src/java/org/apache/solr/cli/DeleteTool.java
@@ -26,7 +26,6 @@
 import java.util.Set;
 import java.util.concurrent.TimeUnit;
 import org.apache.commons.cli.CommandLine;
-import org.apache.commons.cli.DeprecatedAttributes;
 import org.apache.commons.cli.Option;
 import org.apache.solr.client.solrj.SolrClient;
 import org.apache.solr.client.solrj.SolrServerException;
@@ -80,19 +79,6 @@ public List<Option> getOptions() {
             .required(true)
             .desc("Name of the core / collection to delete.")
             .build(),
-        Option.builder("d")
-            .deprecated(
-                DeprecatedAttributes.builder()
-                    .setForRemoval(true)
-                    .setSince("9.8")
-                    .setDescription("Use --delete-config instead")
-                    .get())
-            .hasArg()
-            .argName("true|false")
-            .required(false)
-            .desc(
-                "Flag to indicate if the underlying configuration directory for a collection should also be deleted; default is true.")
-            .build(),
         Option.builder()
             .longOpt("delete-config")
             .hasArg()
@@ -101,44 +87,6 @@ public List<Option> getOptions() {
             .desc(
                 "Flag to indicate if the underlying configuration directory for a collection should also be deleted; default is true.")
             .build(),
-        Option.builder()
-            .longOpt("deleteConfig")
-            .deprecated(
-                DeprecatedAttributes.builder()
-                    .setForRemoval(true)
-                    .setSince("9.7")
-                    .setDescription("Use --delete-config instead")
-                    .get())
-            .hasArg()
-            .argName("true|false")
-            .required(false)
-            .desc(
-                "Flag to indicate if the underlying configuration directory for a collection should also be deleted; default is true.")
-            .build(),
-        Option.builder()
-            .longOpt("force-delete-config")
-            .deprecated(
-                DeprecatedAttributes.builder()
-                    .setForRemoval(true)
-                    .setSince("9.8")
-                    .setDescription("Use --force instead")
-                    .get())
-            .required(false)
-            .desc(
-                "Skip safety checks when deleting the configuration directory used by a collection.")
-            .build(),
-        Option.builder()
-            .longOpt("forceDeleteConfig")
-            .deprecated(
-                DeprecatedAttributes.builder()
-                    .setForRemoval(true)
-                    .setSince("9.7")
-                    .setDescription("Use --force instead")
-                    .get())
-            .required(false)
-            .desc(
-                "Skip safety checks when deleting the configuration directory used by a collection.")
-            .build(),
         Option.builder("f")
             .longOpt("force")
             .required(false)
@@ -146,10 +94,7 @@ public List<Option> getOptions() {
                 "Skip safety checks when deleting the configuration directory used by a collection.")
             .build(),
         SolrCLI.OPTION_SOLRURL,
-        SolrCLI.OPTION_SOLRURL_DEPRECATED,
-        SolrCLI.OPTION_SOLRURL_DEPRECATED_SHORT,
         SolrCLI.OPTION_ZKHOST,
-        SolrCLI.OPTION_ZKHOST_DEPRECATED,
         SolrCLI.OPTION_CREDENTIALS);
   }
 
@@ -201,16 +146,10 @@ protected void deleteCollection(CloudSolrClient cloudSolrClient, CommandLine cli
     boolean deleteConfig = true;
     if (cli.hasOption("delete-config")) {
       deleteConfig = "true".equals(cli.getOptionValue("delete-config"));
-    } else if (cli.hasOption("d")) {
-      deleteConfig = "true".equals(cli.getOptionValue("d"));
-    } else if (cli.hasOption("deleteConfig")) {
-      deleteConfig = "true".equals(cli.getOptionValue("deleteConfig"));
     }
 
     if (deleteConfig && configName != null) {
-      if (cli.hasOption("force")
-          || cli.hasOption("force-delete-config")
-          || cli.hasOption("forceDeleteConfig")) {
+      if (cli.hasOption("force")) {
         log.warn(
             "Skipping safety checks, configuration directory {} will be deleted with impunity.",
             configName);
diff --git a/solr/core/src/java/org/apache/solr/cli/ExportTool.java b/solr/core/src/java/org/apache/solr/cli/ExportTool.java
index 97e4774f391..699df52d330 100644
--- a/solr/core/src/java/org/apache/solr/cli/ExportTool.java
+++ b/solr/core/src/java/org/apache/solr/cli/ExportTool.java
@@ -54,7 +54,6 @@
 import java.util.function.Consumer;
 import java.util.zip.GZIPOutputStream;
 import org.apache.commons.cli.CommandLine;
-import org.apache.commons.cli.DeprecatedAttributes;
 import org.apache.commons.cli.Option;
 import org.apache.lucene.util.SuppressForbidden;
 import org.apache.solr.client.solrj.SolrClient;
@@ -97,36 +96,12 @@ public String getName() {
   @Override
   public List<Option> getOptions() {
     return List.of(
-        Option.builder("url")
-            .longOpt("solr-collection-url")
-            .deprecated(
-                DeprecatedAttributes.builder()
-                    .setForRemoval(true)
-                    .setSince("9.8")
-                    .setDescription("Use --solr-url and -c / --name instead")
-                    .get())
-            .hasArg()
-            .argName("URL")
-            .desc("Address of the collection, example http://localhost:8983/solr/gettingstarted.")
-            .build(),
         Option.builder("c")
             .longOpt("name")
             .hasArg()
             .argName("NAME")
             .desc("Name of the collection.")
             .build(),
-        Option.builder("out")
-            .deprecated(
-                DeprecatedAttributes.builder()
-                    .setForRemoval(true)
-                    .setSince("9.8")
-                    .setDescription("Use --output instead")
-                    .get())
-            .hasArg()
-            .argName("PATH")
-            .desc(
-                "Path to output the exported data, and optionally the file name, defaults to 'collection-name'.")
-            .build(),
         Option.builder()
             .longOpt("output")
             .hasArg()
@@ -291,10 +266,8 @@ public void runImpl(CommandLine cli) throws Exception {
       }
       url = SolrCLI.normalizeSolrUrl(cli) + "/solr/" + cli.getOptionValue("name");
 
-    } else if (cli.hasOption("solr-collection-url")) {
-      url = cli.getOptionValue("solr-collection-url");
     } else {
-      // Swap to required Option when --solr-collection-url removed.
+      // think about support --zk-host someday.
       throw new IllegalArgumentException("Must specify --solr-url.");
     }
     String credentials = cli.getOptionValue(SolrCLI.OPTION_CREDENTIALS.getLongOpt());
@@ -302,9 +275,7 @@ public void runImpl(CommandLine cli) throws Exception {
     info.query = cli.getOptionValue("query", "*:*");
 
     info.setOutFormat(
-        SolrCLI.getOptionWithDeprecatedAndDefault(cli, "output", "out", null),
-        cli.getOptionValue("format"),
-        cli.hasOption("compress"));
+        cli.getOptionValue("output"), cli.getOptionValue("format"), cli.hasOption("compress"));
     info.fields = cli.getOptionValue("fields");
     info.setLimit(cli.getOptionValue("limit", "100"));
     info.output = super.stdout;
diff --git a/solr/core/src/java/org/apache/solr/cli/HealthcheckTool.java b/solr/core/src/java/org/apache/solr/cli/HealthcheckTool.java
index 58024d47ac7..dcc3aa91d11 100644
--- a/solr/core/src/java/org/apache/solr/cli/HealthcheckTool.java
+++ b/solr/core/src/java/org/apache/solr/cli/HealthcheckTool.java
@@ -64,10 +64,7 @@ public List<Option> getOptions() {
             .desc("Name of the collection to check.")
             .build(),
         SolrCLI.OPTION_SOLRURL,
-        SolrCLI.OPTION_SOLRURL_DEPRECATED,
-        SolrCLI.OPTION_SOLRURL_DEPRECATED_SHORT,
         SolrCLI.OPTION_ZKHOST,
-        SolrCLI.OPTION_ZKHOST_DEPRECATED,
         SolrCLI.OPTION_CREDENTIALS);
   }
 
diff --git a/solr/core/src/java/org/apache/solr/cli/LinkConfigTool.java b/solr/core/src/java/org/apache/solr/cli/LinkConfigTool.java
index 1e48a890f3b..da553eb735e 100644
--- a/solr/core/src/java/org/apache/solr/cli/LinkConfigTool.java
+++ b/solr/core/src/java/org/apache/solr/cli/LinkConfigTool.java
@@ -64,8 +64,7 @@ public List<Option> getOptions() {
             .required(true)
             .desc("Configset name in ZooKeeper.")
             .build(),
-        SolrCLI.OPTION_ZKHOST,
-        SolrCLI.OPTION_ZKHOST_DEPRECATED);
+        SolrCLI.OPTION_ZKHOST);
   }
 
   @Override
diff --git a/solr/core/src/java/org/apache/solr/cli/PackageTool.java b/solr/core/src/java/org/apache/solr/cli/PackageTool.java
index 07c5487df48..0089e896ad4 100644
--- a/solr/core/src/java/org/apache/solr/cli/PackageTool.java
+++ b/solr/core/src/java/org/apache/solr/cli/PackageTool.java
@@ -27,7 +27,6 @@
 import java.util.List;
 import java.util.Map;
 import org.apache.commons.cli.CommandLine;
-import org.apache.commons.cli.DeprecatedAttributes;
 import org.apache.commons.cli.Option;
 import org.apache.logging.log4j.Level;
 import org.apache.logging.log4j.core.config.Configurator;
@@ -174,10 +173,7 @@ public void runImpl(CommandLine cli) throws Exception {
                           ? PackageUtils.validateCollections(
                               cli.getOptionValue("collections").split(","))
                           : new String[] {};
-                  String[] parameters =
-                      cli.hasOption("param")
-                          ? cli.getOptionValues("param")
-                          : cli.getOptionValues("p");
+                  String[] parameters = cli.getOptionValues("param");
                   packageManager.deploy(
                       packageName,
                       version,
@@ -341,18 +337,6 @@ public List<Option> getOptions() {
             .argName("PARAMS")
             .desc("List of parameters to be used with deploy command.")
             .build(),
-        Option.builder("p")
-            .deprecated(
-                DeprecatedAttributes.builder()
-                    .setForRemoval(true)
-                    .setSince("9.8")
-                    .setDescription("Use --param instead")
-                    .get())
-            .hasArg()
-            .argName("PARAMS")
-            .required(false)
-            .desc("List of parameters to be used with deploy command.")
-            .build(),
         Option.builder()
             .longOpt("update")
             .desc("If a deployment is an update over a previous deployment.")
@@ -368,10 +352,7 @@ public List<Option> getOptions() {
             .desc("Don't prompt for input; accept all default choices, defaults to false.")
             .build(),
         SolrCLI.OPTION_SOLRURL,
-        SolrCLI.OPTION_SOLRURL_DEPRECATED,
-        SolrCLI.OPTION_SOLRURL_DEPRECATED_SHORT,
         SolrCLI.OPTION_ZKHOST,
-        SolrCLI.OPTION_ZKHOST_DEPRECATED,
         SolrCLI.OPTION_CREDENTIALS);
   }
 }
diff --git a/solr/core/src/java/org/apache/solr/cli/PostLogsTool.java b/solr/core/src/java/org/apache/solr/cli/PostLogsTool.java
index 61b07eecb96..7be103ea923 100644
--- a/solr/core/src/java/org/apache/solr/cli/PostLogsTool.java
+++ b/solr/core/src/java/org/apache/solr/cli/PostLogsTool.java
@@ -36,7 +36,6 @@
 import java.util.stream.Collectors;
 import java.util.stream.Stream;
 import org.apache.commons.cli.CommandLine;
-import org.apache.commons.cli.DeprecatedAttributes;
 import org.apache.commons.cli.Option;
 import org.apache.solr.client.solrj.SolrClient;
 import org.apache.solr.client.solrj.impl.Http2SolrClient;
@@ -65,21 +64,10 @@ public String getName() {
   @Override
   public List<Option> getOptions() {
     return List.of(
-        Option.builder("url")
-            .longOpt("solr-collection-url")
-            .deprecated(
-                DeprecatedAttributes.builder()
-                    .setForRemoval(true)
-                    .setSince("9.8")
-                    .setDescription("Use --solr-url and -c / --name instead")
-                    .get())
-            .hasArg()
-            .argName("ADDRESS")
-            .desc("Address of the collection, example http://localhost:8983/solr/collection1/.")
-            .build(),
         Option.builder("c")
             .longOpt("name")
             .hasArg()
+            .required(true)
             .argName("NAME")
             .desc("Name of the collection.")
             .build(),
@@ -98,16 +86,10 @@ public List<Option> getOptions() {
   public void runImpl(CommandLine cli) throws Exception {
     String url = null;
     if (cli.hasOption("solr-url")) {
-      if (!cli.hasOption("name")) {
-        throw new IllegalArgumentException(
-            "Must specify -c / --name parameter with --solr-url to post documents.");
-      }
       url = SolrCLI.normalizeSolrUrl(cli) + "/solr/" + cli.getOptionValue("name");
 
-    } else if (cli.hasOption("solr-collection-url")) {
-      url = cli.getOptionValue("solr-collection-url");
     } else {
-      // Swap to required Option when --solr-collection-url removed.
+      // Could be required arg, but maybe we want to support --zk-host option too?
       throw new IllegalArgumentException("Must specify --solr-url.");
     }
     String rootDir = cli.getOptionValue("rootdir");
diff --git a/solr/core/src/java/org/apache/solr/cli/PostTool.java b/solr/core/src/java/org/apache/solr/cli/PostTool.java
index d479a90aaba..b888fdca589 100644
--- a/solr/core/src/java/org/apache/solr/cli/PostTool.java
+++ b/solr/core/src/java/org/apache/solr/cli/PostTool.java
@@ -67,7 +67,6 @@
 import javax.xml.xpath.XPathExpressionException;
 import javax.xml.xpath.XPathFactory;
 import org.apache.commons.cli.CommandLine;
-import org.apache.commons.cli.DeprecatedAttributes;
 import org.apache.commons.cli.Option;
 import org.apache.commons.io.output.NullOutputStream;
 import org.apache.solr.client.api.util.SolrVersion;
@@ -172,21 +171,10 @@ public String getName() {
   @Override
   public List<Option> getOptions() {
     return List.of(
-        Option.builder("url")
-            .longOpt("solr-update-url")
-            .deprecated(
-                DeprecatedAttributes.builder()
-                    .setForRemoval(true)
-                    .setSince("9.8")
-                    .setDescription("Use --solr-url and -c / --name instead")
-                    .get())
-            .hasArg()
-            .argName("UPDATEURL")
-            .desc("Solr Update URL, the full url to the update handler, including the /update.")
-            .build(),
         Option.builder("c")
             .longOpt("name")
             .hasArg()
+            .required(true)
             .argName("NAME")
             .desc("Name of the collection.")
             .build(),
@@ -194,16 +182,6 @@ public List<Option> getOptions() {
             .longOpt("skip-commit")
             .desc("Do not 'commit', and thus changes won't be visible till a commit occurs.")
             .build(),
-        Option.builder()
-            .longOpt("skipcommit")
-            .deprecated(
-                DeprecatedAttributes.builder()
-                    .setForRemoval(true)
-                    .setSince("9.7")
-                    .setDescription("Use --skip-commit instead")
-                    .get())
-            .desc("Do not 'commit', and thus changes won't be visible till a commit occurs.")
-            .build(),
         Option.builder("o")
             .longOpt("optimize")
             .desc("Issue an optimize at end of posting documents.")
@@ -222,19 +200,6 @@ public List<Option> getOptions() {
             .required(false)
             .desc("For web crawl, how deep to go. default: 1")
             .build(),
-        Option.builder("d")
-            .deprecated(
-                DeprecatedAttributes.builder()
-                    .setForRemoval(true)
-                    .setSince("9.8")
-                    .setDescription("Use --delay instead")
-                    .get())
-            .hasArg()
-            .argName("delay")
-            .required(false)
-            .desc(
-                "If recursive then delay will be the wait time between posts.  default: 10 for web, 0 for files")
-            .build(),
         Option.builder()
             .longOpt("delay")
             .hasArg()
@@ -264,40 +229,6 @@ public List<Option> getOptions() {
             .required(false)
             .desc("Values must be URL-encoded; these pass through to Solr update request.")
             .build(),
-        Option.builder("p")
-            .deprecated(
-                DeprecatedAttributes.builder()
-                    .setForRemoval(true)
-                    .setSince("9.8")
-                    .setDescription("Use --params instead")
-                    .get())
-            .hasArg()
-            .argName("<key>=<value>[&<key>=<value>...]")
-            .required(false)
-            .desc("Values must be URL-encoded; these pass through to Solr update request.")
-            .build(),
-        Option.builder()
-            .longOpt("out")
-            .deprecated(
-                DeprecatedAttributes.builder()
-                    .setForRemoval(true)
-                    .setSince("9.8")
-                    .setDescription("Use --verbose instead")
-                    .get())
-            .required(false)
-            .desc("sends Solr response outputs to console.")
-            .build(),
-        Option.builder("f")
-            .deprecated(
-                DeprecatedAttributes.builder()
-                    .setForRemoval(true)
-                    .setSince("9.8")
-                    .setDescription("Use --format instead")
-                    .get())
-            .required(false)
-            .desc(
-                "sends application/json content as Solr commands to /update instead of /update/json/docs.")
-            .build(),
         Option.builder()
             .longOpt("format")
             .required(false)
@@ -320,23 +251,13 @@ public void runImpl(CommandLine cli) throws Exception {
 
     solrUpdateUrl = null;
     if (cli.hasOption("solr-url")) {
-      if (!cli.hasOption("name")) {
-        throw new IllegalArgumentException(
-            "Must specify -c / --name parameter with --solr-url to post documents.");
-      }
       String url =
           SolrCLI.normalizeSolrUrl(cli) + "/solr/" + cli.getOptionValue("name") + "/update";
       solrUpdateUrl = new URI(url);
 
-    } else if (cli.hasOption("solr-update-url")) {
-      String url = cli.getOptionValue("solr-update-url");
-      solrUpdateUrl = new URI(url);
-    } else if (cli.hasOption("name")) {
+    } else {
       String url = SolrCLI.getDefaultSolrUrl() + "/solr/" + cli.getOptionValue("name") + "/update";
       solrUpdateUrl = new URI(url);
-    } else {
-      throw new IllegalArgumentException(
-          "Must specify either --solr-update-url or -c parameter to post documents.");
     }
 
     String mode = cli.getOptionValue("mode", DATA_MODE_FILES);
@@ -348,10 +269,7 @@ public void runImpl(CommandLine cli) throws Exception {
       // Turn off automatically looking up the mimetype in favour of what is passed in.
       auto = false;
     }
-    format =
-        cli.hasOption("format") || cli.hasOption("f")
-            ? FORMAT_SOLR
-            : ""; // i.e not solr formatted json commands
+    format = cli.hasOption("format") ? FORMAT_SOLR : ""; // i.e not solr formatted json commands
 
     if (cli.hasOption("filetypes")) {
       fileTypes = cli.getOptionValue("filetypes");
@@ -360,24 +278,19 @@ public void runImpl(CommandLine cli) throws Exception {
     delay = (mode.equals((DATA_MODE_WEB)) ? 10 : 0);
     if (cli.hasOption("delay")) {
       delay = Integer.parseInt(cli.getOptionValue("delay"));
-    } else if (cli.hasOption("d")) {
-      delay = Integer.parseInt(cli.getOptionValue("d"));
     }
 
     recursive = Integer.parseInt(cli.getOptionValue("recursive", "1"));
 
-    out =
-        cli.hasOption("out") || cli.hasOption(SolrCLI.OPTION_VERBOSE.getLongOpt())
-            ? CLIO.getOutStream()
-            : null;
-    commit = !(cli.hasOption("skipcommit") || cli.hasOption("skip-commit"));
+    out = cli.hasOption(SolrCLI.OPTION_VERBOSE.getLongOpt()) ? CLIO.getOutStream() : null;
+    commit = !cli.hasOption("skip-commit");
     optimize = cli.hasOption("optimize");
 
     credentials = cli.getOptionValue(SolrCLI.OPTION_CREDENTIALS.getLongOpt());
 
     args = cli.getArgs();
 
-    params = SolrCLI.getOptionWithDeprecatedAndDefault(cli, "params", "p", "");
+    params = cli.getOptionValue("params", "");
 
     execute(mode);
   }
diff --git a/solr/core/src/java/org/apache/solr/cli/RunExampleTool.java b/solr/core/src/java/org/apache/solr/cli/RunExampleTool.java
index 95c746f82b7..51f1d634703 100644
--- a/solr/core/src/java/org/apache/solr/cli/RunExampleTool.java
+++ b/solr/core/src/java/org/apache/solr/cli/RunExampleTool.java
@@ -36,7 +36,6 @@
 import java.util.Set;
 import java.util.concurrent.TimeUnit;
 import org.apache.commons.cli.CommandLine;
-import org.apache.commons.cli.DeprecatedAttributes;
 import org.apache.commons.cli.Option;
 import org.apache.commons.exec.DefaultExecuteResultHandler;
 import org.apache.commons.exec.DefaultExecutor;
@@ -100,18 +99,6 @@ public List<Option> getOptions() {
             .desc(
                 "Don't prompt for input; accept all defaults when running examples that accept user input.")
             .build(),
-        Option.builder()
-            .longOpt("noprompt")
-            .deprecated(
-                DeprecatedAttributes.builder()
-                    .setForRemoval(true)
-                    .setSince("9.7")
-                    .setDescription("Use --no-prompt instead")
-                    .get())
-            .required(false)
-            .desc(
-                "Don't prompt for input; accept all defaults when running examples that accept user input.")
-            .build(),
         Option.builder("e")
             .longOpt("example")
             .hasArg()
@@ -188,8 +175,7 @@ public List<Option> getOptions() {
             .desc(
                 "Additional options to be passed to the JVM when starting example Solr server(s).")
             .build(),
-        SolrCLI.OPTION_ZKHOST,
-        SolrCLI.OPTION_ZKHOST_DEPRECATED);
+        SolrCLI.OPTION_ZKHOST);
   }
 
   @Override
@@ -447,7 +433,7 @@ protected void runExample(CommandLine cli, String exampleName) throws Exception
 
   protected void runCloudExample(CommandLine cli) throws Exception {
 
-    boolean prompt = !(cli.hasOption("no-prompt") || cli.hasOption("noprompt"));
+    boolean prompt = !cli.hasOption("no-prompt");
     int numNodes = 2;
     int[] cloudPorts = new int[] {8983, 7574, 8984, 7575};
     int defaultPort =
diff --git a/solr/core/src/java/org/apache/solr/cli/SnapshotCreateTool.java b/solr/core/src/java/org/apache/solr/cli/SnapshotCreateTool.java
index 11ff8a205e3..d80dfbc61ea 100644
--- a/solr/core/src/java/org/apache/solr/cli/SnapshotCreateTool.java
+++ b/solr/core/src/java/org/apache/solr/cli/SnapshotCreateTool.java
@@ -45,7 +45,6 @@ public List<Option> getOptions() {
     return List.of(
         SolrCLI.OPTION_ZKHOST,
         SolrCLI.OPTION_SOLRURL,
-        SolrCLI.OPTION_SOLRURL_DEPRECATED_SHORT,
         Option.builder("c")
             .longOpt("name")
             .argName("NAME")
diff --git a/solr/core/src/java/org/apache/solr/cli/SnapshotDeleteTool.java b/solr/core/src/java/org/apache/solr/cli/SnapshotDeleteTool.java
index 8529a35d2f0..d172391183e 100644
--- a/solr/core/src/java/org/apache/solr/cli/SnapshotDeleteTool.java
+++ b/solr/core/src/java/org/apache/solr/cli/SnapshotDeleteTool.java
@@ -45,7 +45,6 @@ public List<Option> getOptions() {
     return List.of(
         SolrCLI.OPTION_ZKHOST,
         SolrCLI.OPTION_SOLRURL,
-        SolrCLI.OPTION_SOLRURL_DEPRECATED_SHORT,
         Option.builder("c")
             .longOpt("name")
             .argName("NAME")
diff --git a/solr/core/src/java/org/apache/solr/cli/SnapshotDescribeTool.java b/solr/core/src/java/org/apache/solr/cli/SnapshotDescribeTool.java
index b8673a6579a..a007b75f075 100644
--- a/solr/core/src/java/org/apache/solr/cli/SnapshotDescribeTool.java
+++ b/solr/core/src/java/org/apache/solr/cli/SnapshotDescribeTool.java
@@ -58,7 +58,6 @@ public List<Option> getOptions() {
     return List.of(
         SolrCLI.OPTION_ZKHOST,
         SolrCLI.OPTION_SOLRURL,
-        SolrCLI.OPTION_SOLRURL_DEPRECATED_SHORT,
         Option.builder("c")
             .longOpt("name")
             .argName("NAME")
diff --git a/solr/core/src/java/org/apache/solr/cli/SnapshotExportTool.java b/solr/core/src/java/org/apache/solr/cli/SnapshotExportTool.java
index 9cc5668d6c3..8471ec79287 100644
--- a/solr/core/src/java/org/apache/solr/cli/SnapshotExportTool.java
+++ b/solr/core/src/java/org/apache/solr/cli/SnapshotExportTool.java
@@ -20,7 +20,6 @@
 import java.util.List;
 import java.util.Optional;
 import org.apache.commons.cli.CommandLine;
-import org.apache.commons.cli.DeprecatedAttributes;
 import org.apache.commons.cli.Option;
 import org.apache.solr.client.solrj.SolrClient;
 import org.apache.solr.client.solrj.request.CollectionAdminRequest;
@@ -47,7 +46,6 @@ public List<Option> getOptions() {
     return List.of(
         SolrCLI.OPTION_ZKHOST,
         SolrCLI.OPTION_SOLRURL,
-        SolrCLI.OPTION_SOLRURL_DEPRECATED_SHORT,
         Option.builder("c")
             .longOpt("name")
             .argName("NAME")
@@ -78,19 +76,6 @@ public List<Option> getOptions() {
             .desc(
                 "Specifies name of the backup repository to be used during snapshot export preparation.")
             .build(),
-        Option.builder("i")
-            .deprecated(
-                DeprecatedAttributes.builder()
-                    .setForRemoval(true)
-                    .setSince("9.8")
-                    .setDescription("Use --async-id instead")
-                    .get())
-            .argName("ID")
-            .hasArg()
-            .required(false)
-            .desc(
-                "Specifies the async request identifier to be used during snapshot export preparation.")
-            .build(),
         Option.builder()
             .longOpt("async-id")
             .argName("ID")
@@ -111,9 +96,6 @@ public void runImpl(CommandLine cli) throws Exception {
     String destDir = cli.getOptionValue("dest-dir");
     Optional<String> backupRepo = Optional.ofNullable(cli.getOptionValue("backup-repo-name"));
     Optional<String> asyncReqId = Optional.ofNullable(cli.getOptionValue("async-id"));
-    if (cli.hasOption("i")) {
-      asyncReqId = Optional.ofNullable(cli.getOptionValue("i"));
-    }
 
     try (var solrClient = SolrCLI.getSolrClient(cli)) {
       exportSnapshot(solrClient, collectionName, snapshotName, destDir, backupRepo, asyncReqId);
diff --git a/solr/core/src/java/org/apache/solr/cli/SnapshotListTool.java b/solr/core/src/java/org/apache/solr/cli/SnapshotListTool.java
index 5ccaf6d6cc5..a1db955b0c0 100644
--- a/solr/core/src/java/org/apache/solr/cli/SnapshotListTool.java
+++ b/solr/core/src/java/org/apache/solr/cli/SnapshotListTool.java
@@ -47,7 +47,6 @@ public List<Option> getOptions() {
     return List.of(
         SolrCLI.OPTION_ZKHOST,
         SolrCLI.OPTION_SOLRURL,
-        SolrCLI.OPTION_SOLRURL_DEPRECATED_SHORT,
         Option.builder("c")
             .longOpt("name")
             .argName("NAME")
diff --git a/solr/core/src/java/org/apache/solr/cli/SolrCLI.java b/solr/core/src/java/org/apache/solr/cli/SolrCLI.java
index 98c22c7f6d8..9f762459fcb 100755
--- a/solr/core/src/java/org/apache/solr/cli/SolrCLI.java
+++ b/solr/core/src/java/org/apache/solr/cli/SolrCLI.java
@@ -46,7 +46,6 @@
 import java.util.zip.ZipInputStream;
 import org.apache.commons.cli.CommandLine;
 import org.apache.commons.cli.DefaultParser;
-import org.apache.commons.cli.DeprecatedAttributes;
 import org.apache.commons.cli.HelpFormatter;
 import org.apache.commons.cli.Option;
 import org.apache.commons.cli.Options;
@@ -88,24 +87,6 @@ public class SolrCLI implements CLIO {
 
   public static final String ZK_HOST = "localhost:9983";
 
-  public static final Option OPTION_ZKHOST_DEPRECATED =
-      Option.builder("zkHost")
-          .longOpt("zkHost")
-          .deprecated(
-              DeprecatedAttributes.builder()
-                  .setForRemoval(true)
-                  .setSince("9.7")
-                  .setDescription("Use --zk-host instead")
-                  .get())
-          .argName("HOST")
-          .hasArg()
-          .required(false)
-          .desc(
-              "Zookeeper connection string; unnecessary if ZK_HOST is defined in solr.in.sh; otherwise, defaults to "
-                  + ZK_HOST
-                  + '.')
-          .build();
-
   public static final Option OPTION_ZKHOST =
       Option.builder("z")
           .longOpt("zk-host")
@@ -117,23 +98,6 @@ public class SolrCLI implements CLIO {
                   + ZK_HOST
                   + '.')
           .build();
-  public static final Option OPTION_SOLRURL_DEPRECATED =
-      Option.builder("solrUrl")
-          .longOpt("solrUrl")
-          .deprecated(
-              DeprecatedAttributes.builder()
-                  .setForRemoval(true)
-                  .setSince("9.7")
-                  .setDescription("Use --solr-url instead")
-                  .get())
-          .argName("HOST")
-          .hasArg()
-          .required(false)
-          .desc(
-              "Base Solr URL, which can be used to determine the zk-host if that's not known; defaults to: "
-                  + getDefaultSolrUrl()
-                  + '.')
-          .build();
   public static final Option OPTION_SOLRURL =
       Option.builder("s")
           .longOpt("solr-url")
@@ -145,22 +109,6 @@ public class SolrCLI implements CLIO {
                   + getDefaultSolrUrl()
                   + '.')
           .build();
-  public static final Option OPTION_SOLRURL_DEPRECATED_SHORT =
-      Option.builder("url")
-          .argName("HOST")
-          .hasArg()
-          .required(false)
-          .deprecated(
-              DeprecatedAttributes.builder()
-                  .setForRemoval(true)
-                  .setSince("9.7")
-                  .setDescription("Use -s instead")
-                  .get())
-          .desc(
-              "Base Solr URL, which can be used to determine the zk-host if that's not known; defaults to: "
-                  + getDefaultSolrUrl()
-                  + '.')
-          .build();
 
   public static final Option OPTION_VERBOSE =
       Option.builder()
@@ -172,19 +120,6 @@ public class SolrCLI implements CLIO {
   public static final Option OPTION_HELP =
       Option.builder("h").longOpt("help").required(false).desc("Print this message.").build();
 
-  public static final Option OPTION_RECURSE_DEPRECATED =
-      Option.builder()
-          .longOpt("recurse")
-          .deprecated(
-              DeprecatedAttributes.builder()
-                  .setForRemoval(true)
-                  .setSince("9.8")
-                  .setDescription("Use --recursive instead")
-                  .get())
-          .required(false)
-          .desc("Apply the command recursively.")
-          .build();
-
   public static final Option OPTION_RECURSIVE =
       Option.builder("r")
           .longOpt("recursive")
@@ -215,19 +150,13 @@ public static void exit(int exitStatus) {
   public static void main(String[] args) throws Exception {
     final boolean hasNoCommand =
         args == null || args.length == 0 || args[0] == null || args[0].trim().length() == 0;
-    final boolean isHelpCommand =
-        !hasNoCommand && Arrays.asList("-h", "--help", "/?").contains(args[0]);
+    final boolean isHelpCommand = !hasNoCommand && Arrays.asList("-h", "--help").contains(args[0]);
 
     if (hasNoCommand || isHelpCommand) {
       printHelp();
       exit(1);
     }
 
-    if (Arrays.asList("-version", "version").contains(args[0])) {
-      // select the version tool to be run
-      CLIO.out("Deprecated operation as of 9.8.  Please use bin/solr --version.");
-      args = new String[] {"version"};
-    }
     if (Arrays.asList("-v", "--version").contains(args[0])) {
       // select the version tool to be run
       args = new String[] {"version"};
@@ -243,7 +172,7 @@ public static void main(String[] args) throws Exception {
         // remap our arguments to invoke the ZK tool help.
         args = new String[] {"zk-tool-help", "--print-long-zk-usage"};
       } else if (args.length == 2) {
-        if (Arrays.asList("-h", "--help", "/?").contains(args[1])) {
+        if (Arrays.asList("-h", "--help").contains(args[1])) {
           // remap our arguments to invoke the ZK tool help.
           args = new String[] {"zk-tool-help", "--print-long-zk-usage"};
         } else {
@@ -411,6 +340,10 @@ public static Options getToolOptions(Tool tool) {
   /**
    * Returns the value of the option with the given name, or the value of the deprecated option. If
    * both values are null, then it returns the default value.
+   *
+   * <p>If this method is marked as unused by your IDE, it means we have no deprecated CLI options
+   * currently, congratulations! This method is preserved for the next time we need to deprecate a
+   * CLI option.
    */
   public static String getOptionWithDeprecatedAndDefault(
       CommandLine cli, String opt, String deprecated, String def) {
@@ -462,7 +395,7 @@ public static CommandLine processCommandLineArgs(Tool tool, String[] args) {
       boolean hasHelpArg = false;
       if (args != null) {
         for (String arg : args) {
-          if ("-h".equals(arg) || "--help".equals(arg) || "-help".equals(arg)) {
+          if ("-h".equals(arg) || "--help".equals(arg)) {
             hasHelpArg = true;
             break;
           }
@@ -739,18 +672,10 @@ public static String normalizeSolrUrl(String solrUrl, boolean logUrlFormatWarnin
    * ZooKeeper.
    */
   public static String normalizeSolrUrl(CommandLine cli) throws Exception {
-    String solrUrl = null;
-    if (cli.hasOption("solr-url")) {
-      solrUrl = cli.getOptionValue("solr-url");
-    } else if (cli.hasOption("solrUrl")) {
-      solrUrl = cli.getOptionValue("solrUrl");
-    } else if (cli.hasOption("url")) {
-      solrUrl = cli.getOptionValue("url");
-    }
+    String solrUrl = cli.getOptionValue("solr-url");
 
     if (solrUrl == null) {
-      String zkHost =
-          cli.hasOption("zk-host") ? cli.getOptionValue("zk-host") : cli.getOptionValue("zkHost");
+      String zkHost = cli.getOptionValue("zk-host");
       if (zkHost == null) {
         solrUrl = SolrCLI.getDefaultSolrUrl();
         CLIO.err(
@@ -781,8 +706,7 @@ public static String normalizeSolrUrl(CommandLine cli) throws Exception {
    */
   public static String getZkHost(CommandLine cli) throws Exception {
 
-    String zkHost =
-        cli.hasOption("zk-host") ? cli.getOptionValue("zk-host") : cli.getOptionValue("zkHost");
+    String zkHost = cli.getOptionValue("zk-host");
     if (zkHost != null && !zkHost.isBlank()) {
       return zkHost;
     }
diff --git a/solr/core/src/java/org/apache/solr/cli/StatusTool.java b/solr/core/src/java/org/apache/solr/cli/StatusTool.java
index 94f46b106dd..16ad16e5946 100644
--- a/solr/core/src/java/org/apache/solr/cli/StatusTool.java
+++ b/solr/core/src/java/org/apache/solr/cli/StatusTool.java
@@ -236,7 +236,7 @@ public boolean waitForSolrUp(String solrUrl, CommandLine cli, int maxWaitSecs) t
     }
   }
 
-  public boolean printStatusFromRunningSolr(String solrUrl, CommandLine cli) throws Exception {
+  public boolean printStatusFromRunningSolr(String solrUrl, CommandLine cli) {
     String statusJson = null;
     try {
       statusJson = statusFromRunningSolr(solrUrl, cli);
diff --git a/solr/core/src/java/org/apache/solr/cli/UpdateACLTool.java b/solr/core/src/java/org/apache/solr/cli/UpdateACLTool.java
index f98959a89c7..e9a9c5b444e 100644
--- a/solr/core/src/java/org/apache/solr/cli/UpdateACLTool.java
+++ b/solr/core/src/java/org/apache/solr/cli/UpdateACLTool.java
@@ -54,7 +54,7 @@ public String getUsage() {
 
   @Override
   public List<Option> getOptions() {
-    return List.of(SolrCLI.OPTION_ZKHOST, SolrCLI.OPTION_ZKHOST_DEPRECATED);
+    return List.of(SolrCLI.OPTION_ZKHOST);
   }
 
   @Override
diff --git a/solr/core/src/java/org/apache/solr/cli/ZkCpTool.java b/solr/core/src/java/org/apache/solr/cli/ZkCpTool.java
index 197156c8652..488618e2f41 100644
--- a/solr/core/src/java/org/apache/solr/cli/ZkCpTool.java
+++ b/solr/core/src/java/org/apache/solr/cli/ZkCpTool.java
@@ -62,13 +62,9 @@ public List<Option> getOptions() {
             .required(false)
             .desc("Required to look up configuration for compressing state.json.")
             .build(),
-        SolrCLI.OPTION_RECURSE_DEPRECATED,
         SolrCLI.OPTION_RECURSIVE,
         SolrCLI.OPTION_SOLRURL,
-        SolrCLI.OPTION_SOLRURL_DEPRECATED,
-        SolrCLI.OPTION_SOLRURL_DEPRECATED_SHORT,
         SolrCLI.OPTION_ZKHOST,
-        SolrCLI.OPTION_ZKHOST_DEPRECATED,
         SolrCLI.OPTION_CREDENTIALS);
   }
 
@@ -133,7 +129,7 @@ public void runImpl(CommandLine cli) throws Exception {
     echoIfVerbose("\nConnecting to ZooKeeper at " + zkHost + " ...");
     String src = cli.getArgs()[0];
     String dst = cli.getArgs()[1];
-    boolean recursive = cli.hasOption("recursive") || cli.hasOption("recurse");
+    boolean recursive = cli.hasOption("recursive");
     echo("Copying from '" + src + "' to '" + dst + "'. ZooKeeper at " + zkHost);
 
     boolean srcIsZk = src.toLowerCase(Locale.ROOT).startsWith("zk:");
diff --git a/solr/core/src/java/org/apache/solr/cli/ZkLsTool.java b/solr/core/src/java/org/apache/solr/cli/ZkLsTool.java
index 329e8f47dc3..cefb37b0e76 100644
--- a/solr/core/src/java/org/apache/solr/cli/ZkLsTool.java
+++ b/solr/core/src/java/org/apache/solr/cli/ZkLsTool.java
@@ -40,13 +40,9 @@ public ZkLsTool(PrintStream stdout) {
   @Override
   public List<Option> getOptions() {
     return List.of(
-        SolrCLI.OPTION_RECURSE_DEPRECATED,
         SolrCLI.OPTION_RECURSIVE,
         SolrCLI.OPTION_SOLRURL,
-        SolrCLI.OPTION_SOLRURL_DEPRECATED,
-        SolrCLI.OPTION_SOLRURL_DEPRECATED_SHORT,
         SolrCLI.OPTION_ZKHOST,
-        SolrCLI.OPTION_ZKHOST_DEPRECATED,
         SolrCLI.OPTION_CREDENTIALS);
   }
 
@@ -70,7 +66,7 @@ public void runImpl(CommandLine cli) throws Exception {
     try (SolrZkClient zkClient = SolrCLI.getSolrZkClient(cli, zkHost)) {
       echoIfVerbose("\nConnecting to ZooKeeper at " + zkHost + " ...");
 
-      boolean recursive = cli.hasOption("recursive") || cli.hasOption("recurse");
+      boolean recursive = cli.hasOption("recursive");
       echoIfVerbose(
           "Getting listing for ZooKeeper node "
               + znode
diff --git a/solr/core/src/java/org/apache/solr/cli/ZkMkrootTool.java b/solr/core/src/java/org/apache/solr/cli/ZkMkrootTool.java
index 3c69b7b0d10..606586d1b4f 100644
--- a/solr/core/src/java/org/apache/solr/cli/ZkMkrootTool.java
+++ b/solr/core/src/java/org/apache/solr/cli/ZkMkrootTool.java
@@ -49,10 +49,7 @@ public List<Option> getOptions() {
             .desc("Raise an error if the root exists.  Defaults to false.")
             .build(),
         SolrCLI.OPTION_SOLRURL,
-        SolrCLI.OPTION_SOLRURL_DEPRECATED,
-        SolrCLI.OPTION_SOLRURL_DEPRECATED_SHORT,
         SolrCLI.OPTION_ZKHOST,
-        SolrCLI.OPTION_ZKHOST_DEPRECATED,
         SolrCLI.OPTION_CREDENTIALS);
   }
 
diff --git a/solr/core/src/java/org/apache/solr/cli/ZkMvTool.java b/solr/core/src/java/org/apache/solr/cli/ZkMvTool.java
index 339d7e41040..f49f1484c4b 100644
--- a/solr/core/src/java/org/apache/solr/cli/ZkMvTool.java
+++ b/solr/core/src/java/org/apache/solr/cli/ZkMvTool.java
@@ -43,13 +43,7 @@ public ZkMvTool(PrintStream stdout) {
 
   @Override
   public List<Option> getOptions() {
-    return List.of(
-        SolrCLI.OPTION_SOLRURL,
-        SolrCLI.OPTION_SOLRURL_DEPRECATED,
-        SolrCLI.OPTION_SOLRURL_DEPRECATED_SHORT,
-        SolrCLI.OPTION_ZKHOST,
-        SolrCLI.OPTION_ZKHOST_DEPRECATED,
-        SolrCLI.OPTION_CREDENTIALS);
+    return List.of(SolrCLI.OPTION_SOLRURL, SolrCLI.OPTION_ZKHOST, SolrCLI.OPTION_CREDENTIALS);
   }
 
   @Override
diff --git a/solr/core/src/java/org/apache/solr/cli/ZkRmTool.java b/solr/core/src/java/org/apache/solr/cli/ZkRmTool.java
index bf3e408ca53..4f4e1bf62e6 100644
--- a/solr/core/src/java/org/apache/solr/cli/ZkRmTool.java
+++ b/solr/core/src/java/org/apache/solr/cli/ZkRmTool.java
@@ -42,13 +42,9 @@ public ZkRmTool(PrintStream stdout) {
   @Override
   public List<Option> getOptions() {
     return List.of(
-        SolrCLI.OPTION_RECURSE_DEPRECATED,
         SolrCLI.OPTION_RECURSIVE,
         SolrCLI.OPTION_SOLRURL,
-        SolrCLI.OPTION_SOLRURL_DEPRECATED,
-        SolrCLI.OPTION_SOLRURL_DEPRECATED_SHORT,
         SolrCLI.OPTION_ZKHOST,
-        SolrCLI.OPTION_ZKHOST_DEPRECATED,
         SolrCLI.OPTION_CREDENTIALS);
   }
 
@@ -68,7 +64,7 @@ public void runImpl(CommandLine cli) throws Exception {
     String zkHost = SolrCLI.getZkHost(cli);
 
     String target = cli.getArgs()[0];
-    boolean recursive = cli.hasOption("recursive") || cli.hasOption("recurse");
+    boolean recursive = cli.hasOption("recursive");
 
     String znode = target;
     if (target.toLowerCase(Locale.ROOT).startsWith("zk:")) {
diff --git a/solr/core/src/test/org/apache/solr/cli/CreateToolTest.java b/solr/core/src/test/org/apache/solr/cli/CreateToolTest.java
index 838c3857ce9..7b4da6f9901 100644
--- a/solr/core/src/test/org/apache/solr/cli/CreateToolTest.java
+++ b/solr/core/src/test/org/apache/solr/cli/CreateToolTest.java
@@ -51,7 +51,7 @@ public void testCreateCollectionWithBasicAuth() throws Exception {
       cluster.getZkClient().getZkServerAddress(),
       "--credentials",
       SecurityJson.USER_PASS,
-      "-verbose"
+      "--verbose"
     };
 
     assertEquals(0, runTool(args));
diff --git a/solr/core/src/test/org/apache/solr/cli/DeleteToolTest.java b/solr/core/src/test/org/apache/solr/cli/DeleteToolTest.java
index 86459b1f71a..8f66d104d4e 100644
--- a/solr/core/src/test/org/apache/solr/cli/DeleteToolTest.java
+++ b/solr/core/src/test/org/apache/solr/cli/DeleteToolTest.java
@@ -66,7 +66,7 @@ public void testDeleteCollectionWithBasicAuth() throws Exception {
       cluster.getZkClient().getZkServerAddress(),
       "--credentials",
       SecurityJson.USER_PASS,
-      "-verbose"
+      "--verbose"
     };
     assertEquals(0, runTool(args));
   }
@@ -89,7 +89,7 @@ public void testFailsToDeleteProtectedCollection() throws Exception {
       "testFailsToDeleteProtectedCollection",
       "-z",
       cluster.getZkClient().getZkServerAddress(),
-      "-verbose"
+      "--verbose"
     };
     assertEquals(1, runTool(args));
   }
diff --git a/solr/core/src/test/org/apache/solr/cli/PackageToolTest.java b/solr/core/src/test/org/apache/solr/cli/PackageToolTest.java
index f4155641910..fa75ba3206e 100644
--- a/solr/core/src/test/org/apache/solr/cli/PackageToolTest.java
+++ b/solr/core/src/test/org/apache/solr/cli/PackageToolTest.java
@@ -168,7 +168,7 @@ public void testPackageTool() throws Exception {
           "-y",
           "--collections",
           "abc",
-          "-p",
+          "--param",
           "RH-HANDLER-PATH=" + rhPath,
           "--credentials",
           SecurityJson.USER_PASS
diff --git a/solr/packaging/test/test_bats.bats b/solr/packaging/test/test_bats.bats
index fae2f5be371..631fe2a8f87 100644
--- a/solr/packaging/test/test_bats.bats
+++ b/solr/packaging/test/test_bats.bats
@@ -23,7 +23,7 @@ setup_file() {
   # set up paths and helpers
   common_clean_setup
 
-  solr start -V
+  solr start --verbose
   # echo $output >&3
 }
 
diff --git a/solr/packaging/test/test_config.bats b/solr/packaging/test/test_config.bats
index 243f4fb2689..5400d6d91cb 100644
--- a/solr/packaging/test/test_config.bats
+++ b/solr/packaging/test/test_config.bats
@@ -54,31 +54,21 @@ teardown() {
 @test "short form of setting property" {
   solr create -c COLL_NAME
 
-  run solr config -c COLL_NAME --property updateHandler.autoCommit.maxDocs -v 100
+  run solr config -c COLL_NAME --property updateHandler.autoCommit.maxDocs --value 100
   assert_output --partial "Successfully set-property updateHandler.autoCommit.maxDocs to 100"
-  assert_output --partial "Deprecated for removal since 9.8: Use --value instead"
   assert_output --partial "assuming solr url is http://localhost:${SOLR_PORT}."
 }
 
-# This test is to validate the deprecated and non deprecated options for connecting to Solr
 @test "connecting to solr via various solr urls and zk hosts" {
   solr create -c COLL_NAME
 
-  run solr config -c COLL_NAME --property updateHandler.autoCommit.maxDocs -v 100 -solrUrl http://localhost:${SOLR_PORT}
+  run solr config -c COLL_NAME --property updateHandler.autoCommit.maxDocs --value 100 -s http://localhost:${SOLR_PORT}
   assert_output --partial "Successfully set-property updateHandler.autoCommit.maxDocs to 100"
-  assert_output --partial "Deprecated for removal since 9.7: Use --solr-url instead"
 
-  run solr config -c COLL_NAME --property updateHandler.autoCommit.maxDocs -v 100 --solr-url http://localhost:${SOLR_PORT}
+  run solr config -c COLL_NAME --property updateHandler.autoCommit.maxDocs --value 100 -z localhost:${ZK_PORT}
   assert_output --partial "Successfully set-property updateHandler.autoCommit.maxDocs to 100"
 
-  run solr config -c COLL_NAME --property updateHandler.autoCommit.maxDocs -v 100 -zkHost localhost:${ZK_PORT}
-  assert_output --partial "Successfully set-property updateHandler.autoCommit.maxDocs to 100"
-  assert_output --partial "Deprecated for removal since 9.7: Use --zk-host instead"
-
-  run solr config -c COLL_NAME --property updateHandler.autoCommit.maxDocs -v 100 -z localhost:${ZK_PORT}
-  assert_output --partial "Successfully set-property updateHandler.autoCommit.maxDocs to 100"
-
-  run solr config -c COLL_NAME --property updateHandler.autoCommit.maxDocs -v 100 --zk-host localhost:${ZK_PORT}
+  run solr config -c COLL_NAME --property updateHandler.autoCommit.maxDocs --value 100 --zk-host localhost:${ZK_PORT}
   assert_output --partial "Successfully set-property updateHandler.autoCommit.maxDocs to 100"
 
 }
diff --git a/solr/packaging/test/test_post.bats b/solr/packaging/test/test_post.bats
index 016dba34d36..277631297f9 100644
--- a/solr/packaging/test/test_post.bats
+++ b/solr/packaging/test/test_post.bats
@@ -39,7 +39,7 @@ teardown() {
 @test "Check help command" {
 
   run solr post
-  assert_output --partial 'Must specify either --solr-update-url or -c parameter'
+  assert_output --partial 'Missing required option: c'
 
   run solr post -h
   assert_output --partial 'usage: bin/solr post'
@@ -57,7 +57,7 @@ teardown() {
   run solr create -c monitors -d _default
   assert_output --partial "Created collection 'monitors'"
 
-  run solr post --type application/xml --solr-update-url http://localhost:${SOLR_PORT}/solr/monitors/update ${SOLR_TIP}/example/exampledocs/monitor.xml
+  run solr post --type application/xml --solr-url http://localhost:${SOLR_PORT} -c monitors ${SOLR_TIP}/example/exampledocs/monitor.xml
 
   assert_output --partial '1 files indexed.'
   refute_output --partial 'ERROR'
@@ -89,7 +89,7 @@ teardown() {
 
   solr create -c monitors_no_type -d _default
 
-  run solr post --solr-update-url http://localhost:${SOLR_PORT}/solr/monitors_no_type/update ${SOLR_TIP}/example/exampledocs/monitor.xml
+  run solr post --solr-url http://localhost:${SOLR_PORT} -c monitors_no_type ${SOLR_TIP}/example/exampledocs/monitor.xml
 
   assert_output --partial '1 files indexed.'
   refute_output --partial 'ERROR'
@@ -98,7 +98,7 @@ teardown() {
 
   solr create -c books_no_type -d _default
 
-  run solr post --solr-update-url http://localhost:${SOLR_PORT}/solr/books_no_type/update ${SOLR_TIP}/example/exampledocs/books.json
+  run solr post --solr-url http://localhost:${SOLR_PORT} -c books_no_type ${SOLR_TIP}/example/exampledocs/books.json
 
   assert_output --partial '1 files indexed.'
   refute_output --partial 'ERROR'
@@ -107,7 +107,7 @@ teardown() {
 
   solr create -c books_csv_no_type -d _default
 
-  run solr post --solr-update-url http://localhost:${SOLR_PORT}/solr/books_csv_no_type/update ${SOLR_TIP}/example/exampledocs/books.csv
+  run solr post --solr-url http://localhost:${SOLR_PORT} -c books_csv_no_type ${SOLR_TIP}/example/exampledocs/books.csv
 
   assert_output --partial '1 files indexed.'
   refute_output --partial 'ERROR'
@@ -118,7 +118,7 @@ teardown() {
 @test "crawling a directory as a dry-run" {
 
   # We filter to xml,json,and csv as we don't want to invoke the Extract handler, and are running it as a dry run
-  run solr post --dry-run --filetypes xml,json,csv --solr-update-url http://localhost:${SOLR_PORT}/solr/foobar/update --skip-commit ${SOLR_TIP}/example/exampledocs
+  run solr post --dry-run --filetypes xml,json,csv --solr-url http://localhost:${SOLR_PORT} -c foobar --skip-commit ${SOLR_TIP}/example/exampledocs
 
   assert_output --partial 'Dry run complete. 16 would have been indexed.'
   refute_output --partial '16 files indexed.'
@@ -130,7 +130,7 @@ teardown() {
   solr create -c mixed_content -d _default
 
   # We filter to xml,json,and csv as we don't want to invoke the Extract handler.
-  run solr post --filetypes xml,json,csv --solr-update-url http://localhost:${SOLR_PORT}/solr/mixed_content/update ${SOLR_TIP}/example/exampledocs
+  run solr post --filetypes xml,json,csv --solr-url http://localhost:${SOLR_PORT} -c mixed_content ${SOLR_TIP}/example/exampledocs
 
   assert_output --partial '16 files indexed.'
   refute_output --partial 'ERROR'
@@ -150,7 +150,7 @@ teardown() {
     }
   }' "http://localhost:${SOLR_PORT}/solr/webcrawl/config"
 
-  run solr post --mode web --solr-update-url http://localhost:${SOLR_PORT}/webcrawl/update --recursive 1 --delay 1 https://solr.apache.org
+  run solr post --mode web -c webcrawl --recursive 1 --delay 1 https://solr.apache.org
   assert_output --partial 'Entering crawl at level 0'
 }
 
@@ -159,7 +159,7 @@ teardown() {
   run solr create -c monitors2 -d _default
   assert_output --partial "Created collection 'monitors2'"
 
-  run solr post --solr-update-url http://localhost:${SOLR_PORT}/solr/monitors2/update --type application/xml --skip-commit --optimize ${SOLR_TIP}/example/exampledocs/monitor.xml
+  run solr post --solr-url http://localhost:${SOLR_PORT} -c monitors2 --type application/xml --skip-commit --optimize ${SOLR_TIP}/example/exampledocs/monitor.xml
 
   assert_output --partial '1 files indexed.'
   refute_output --partial 'COMMITting Solr index'
@@ -173,18 +173,18 @@ teardown() {
   run solr create -c test_args -d _default
   assert_output --partial "Created collection 'test_args'"
 
-  run solr post --solr-update-url http://localhost:${SOLR_PORT}/solr/test_args/update --mode args --type application/xml --out "<delete><query>*:*</query></delete>"
+  run solr post --solr-url http://localhost:${SOLR_PORT} -c test_args --mode args --type application/xml --verbose "<delete><query>*:*</query></delete>"
   assert_output --partial '<int name="status">0</int>'
 
   # confirm default type
-  run solr post --solr-update-url http://localhost:${SOLR_PORT}/solr/test_args/update --mode args --out "{'delete': {'query': '*:*'}}"
+  run solr post --solr-url http://localhost:${SOLR_PORT} -c test_args --mode args --verbose "{'delete': {'query': '*:*'}}"
   assert_output --partial '"status":0'
 
-  # confirm we don't get back output without --out
-  run solr post --solr-update-url http://localhost:${SOLR_PORT}/solr/test_args/update --mode args "{'delete': {'query': '*:*'}}"
+  # confirm we don't get back output without --verbose
+  run solr post --solr-url http://localhost:${SOLR_PORT} -c test_args --mode args "{'delete': {'query': '*:*'}}"
   refute_output --partial '"status":0'
 
-  run solr post --solr-update-url http://localhost:${SOLR_PORT}/solr/test_args/update --mode args --type text/csv --out $'id,value\nROW1,0.47'
+  run solr post --solr-url http://localhost:${SOLR_PORT} -c test_args --mode args --type text/csv --verbose $'id,value\nROW1,0.47'
   assert_output --partial '"status":0'
   run curl "http://localhost:${SOLR_PORT}/solr/test_args/select?q=id:ROW1"
   assert_output --partial '"numFound":1'
@@ -192,7 +192,7 @@ teardown() {
 
 # function used because run echo | solr ends up being (run echo) | solr and we loose the output capture.
 capture_echo_to_solr() {
-  echo "{'commit': {}}" | solr post --solr-update-url http://localhost:${SOLR_PORT}/solr/test_stdin/update --mode stdin --type application/json --out
+  echo "{'commit': {}}" | solr post --solr-url http://localhost:${SOLR_PORT} -c test_stdin --mode stdin --type application/json --verbose
 }
 
 @test "stdin mode" {
@@ -209,7 +209,7 @@ capture_echo_to_solr() {
   run solr create -c monitors_verbose -d _default
   assert_output --partial "Created collection 'monitors_verbose'"
 
-  run solr post --verbose --type application/xml --solr-update-url http://localhost:${SOLR_PORT}/solr/monitors_verbose/update ${SOLR_TIP}/example/exampledocs/monitor.xml
+  run solr post --verbose --type application/xml --solr-url http://localhost:${SOLR_PORT} -c monitors_verbose ${SOLR_TIP}/example/exampledocs/monitor.xml
 
   assert_output --partial '1 files indexed.'
   assert_output --partial '<lst name="responseHeader">'
diff --git a/solr/packaging/test/test_postlogs.bats b/solr/packaging/test/test_postlogs.bats
index ef6fe76953f..76893566d43 100644
--- a/solr/packaging/test/test_postlogs.bats
+++ b/solr/packaging/test/test_postlogs.bats
@@ -42,7 +42,7 @@ teardown() {
   run solr create -c COLL_NAME
   assert_output --partial "Created collection 'COLL_NAME'"
 
-  run solr postlogs -url http://localhost:${SOLR_PORT}/solr/COLL_NAME -rootdir ${SOLR_LOGS_DIR}/solr.log
+  run solr postlogs --solr-url http://localhost:${SOLR_PORT} -c COLL_NAME -rootdir ${SOLR_LOGS_DIR}/solr.log
   assert_output --partial 'Sending last batch'
   assert_output --partial 'Committed'
 
diff --git a/solr/packaging/test/test_prometheus.bats b/solr/packaging/test/test_prometheus.bats
index a6dc1c2671d..b9d2689ab79 100644
--- a/solr/packaging/test/test_prometheus.bats
+++ b/solr/packaging/test/test_prometheus.bats
@@ -37,8 +37,8 @@ teardown() {
   run solr create -c COLL_NAME
   assert_output --partial "Created new core 'COLL_NAME'"
  
-  echo "# starting solr-exporter on ${SOLR_EXPORTER_PORT}" >&3
-  run solr-exporter --cluster-id bats-test -p $SOLR_EXPORTER_PORT -b http://localhost:${SOLR_PORT}/solr >&3 &
+  # echo "# starting solr-exporter on ${SOLR_EXPORTER_PORT}" >&3
+  run solr-exporter --cluster-id bats-test -p $SOLR_EXPORTER_PORT --solr-url http://localhost:${SOLR_PORT}/solr >&3 &
 
   sleep 5
 
diff --git a/solr/packaging/test/test_snapshots.bats b/solr/packaging/test/test_snapshots.bats
index c8dd9da8d4c..5bceed84863 100644
--- a/solr/packaging/test/test_snapshots.bats
+++ b/solr/packaging/test/test_snapshots.bats
@@ -26,7 +26,7 @@ setup_file() {
 
 teardown_file() {
   common_setup
-  solr stop -all
+  solr stop --all
 
 }
 
@@ -59,15 +59,16 @@ teardown() {
 @test "snapshot list" {  
   solr snapshot-create -c films --snapshot-name snapshot3 --solr-url http://localhost:${SOLR_PORT}
   
-  run solr snapshot-list -c films -url http://localhost:${SOLR_PORT}/solr
+  # Confirm that we continue to normalize away the /solr ending.
+  run solr snapshot-list -c films -s http://localhost:${SOLR_PORT}/solr
   assert_output --partial "snapshot3"
   
-  run solr snapshot-delete -c films --snapshot-name snapshot3 -url http://localhost:${SOLR_PORT}
+  run solr snapshot-delete -c films --snapshot-name snapshot3 -s http://localhost:${SOLR_PORT}
   assert_output --partial "Successfully deleted snapshot with name snapshot3 for collection films"
 }
 
 @test "snapshot describe" {  
-  solr snapshot-create -c films --snapshot-name snapshot4 -url http://localhost:${SOLR_PORT}
+  solr snapshot-create -c films --snapshot-name snapshot4 -s http://localhost:${SOLR_PORT}
   
   run solr snapshot-describe -c films --snapshot-name snapshot4
   assert_output --partial "Name: snapshot4"
diff --git a/solr/packaging/test/test_ssl.bats b/solr/packaging/test/test_ssl.bats
index 3d13a16b5db..25615a6fd03 100644
--- a/solr/packaging/test/test_ssl.bats
+++ b/solr/packaging/test/test_ssl.bats
@@ -533,8 +533,7 @@ teardown() {
   export SOLR_SSL_KEY_STORE=$ssl_dir/server2.keystore.p12
   export SOLR_SSL_TRUST_STORE=$ssl_dir/server2.keystore.p12
   
-  # leaving -a instead of --jvm-opts for back compat testing.
-  solr start -z localhost:${ZK_PORT} -p ${SOLR2_PORT} -a "-Dsolr.jetty.sslContext.reload.scanInterval=1 -DsocketTimeout=5000"
+  solr start -z localhost:${ZK_PORT} -p ${SOLR2_PORT} --jvm-opts "-Dsolr.jetty.sslContext.reload.scanInterval=1 -DsocketTimeout=5000"
   solr assert --started https://localhost:${SOLR2_PORT} --timeout 5000
 
   # "test" collection is two shards, meaning there must be communication between shards for queries (handled by http shard handler factory)
diff --git a/solr/packaging/test/test_version.bats b/solr/packaging/test/test_version.bats
index edfebd34663..f4d888033e6 100644
--- a/solr/packaging/test/test_version.bats
+++ b/solr/packaging/test/test_version.bats
@@ -21,22 +21,14 @@ setup() {
   common_clean_setup
 }
 
-@test "--version and -v both return Solr version" {
+@test "--version returns Solr version" {
   run solr --version
-  assert_output --partial "Solr version is:"
-  
-  run solr -v
-  assert_output --partial "Solr version is:"
-  
+  assert_output --partial "Solr version is:"    
 }
 
-@test "-version and version both return Solr version and deprecation" {
-  run solr -version
-  assert_output --partial "Solr version is:"
-  assert_output --partial "Deprecated operation as of 9.8.  Please use bin/solr --version."
-  
+@test "version as direct tool call still runs" {  
+  #run ! solr version
+  #assert_output --partial "version is not a valid command!  Did you mean --version?"  
   run solr version
   assert_output --partial "Solr version is:"
-  assert_output --partial "Deprecated operation as of 9.8.  Please use bin/solr --version."
-  
 }
diff --git a/solr/packaging/test/test_zk.bats b/solr/packaging/test/test_zk.bats
index c6a6b184028..240fb072b31 100644
--- a/solr/packaging/test/test_zk.bats
+++ b/solr/packaging/test/test_zk.bats
@@ -66,18 +66,11 @@ teardown() {
 
 @test "connecting to solr via various solr urls and zk hosts" {
   sleep 1
-  run solr zk ls / -solrUrl http://localhost:${SOLR_PORT}
-  assert_output --partial "aliases.json"
-  # We do mapping in bin/solr script from -solrUrl to --solr-url that prevents deprecation warning
-  #assert_output --partial "Deprecated for removal since 9.7: Use --solr-url instead"
-
-  run solr zk ls / -url http://localhost:${SOLR_PORT}
-  assert_output --partial "aliases.json"
-  # We do mapping in bin/solr script from -solrUrl to --solr-url that prevents deprecation warning
-  #assert_output --partial "Deprecated for removal since 9.7: Use --solr-url instead"
-
   run solr zk ls / --solr-url http://localhost:${SOLR_PORT}
   assert_output --partial "aliases.json"
+ 
+  run solr zk ls / -s http://localhost:${SOLR_PORT}
+  assert_output --partial "aliases.json"
 
   run solr zk ls /
   assert_output --partial "aliases.json"
@@ -87,11 +80,6 @@ teardown() {
 
   run solr zk ls / --zk-host localhost:${ZK_PORT}
   assert_output --partial "aliases.json"
-
-  run solr zk ls / -zkHost localhost:${ZK_PORT}
-  assert_output --partial "aliases.json"
-  # We do mapping in bin/solr script from -zkHost to --zk-host that prevents deprecation warning
-  #assert_output --partial "Deprecated for removal since 9.7: Use --zk-host instead"
 }
 
 @test "copying files around" {
diff --git a/solr/prometheus-exporter/bin/solr-exporter b/solr/prometheus-exporter/bin/solr-exporter
index 452e597ab55..20377c4b171 100755
--- a/solr/prometheus-exporter/bin/solr-exporter
+++ b/solr/prometheus-exporter/bin/solr-exporter
@@ -120,9 +120,9 @@ if [[ -n "$NUM_THREADS" ]]; then
 fi
 
 if [[ -n "$ZK_HOST" ]]; then
-  EXPORTER_ARGS+=(-z "$ZK_HOST")
+  EXPORTER_ARGS+=(--zk-host "$ZK_HOST")
 elif [[ -n "$SOLR_URL" ]]; then
-  EXPORTER_ARGS+=(-b "$SOLR_URL")
+  EXPORTER_ARGS+=(--solr-url "$SOLR_URL")
 fi
 
 if [[ -n "$CLUSTER_ID" ]]; then
@@ -130,11 +130,11 @@ if [[ -n "$CLUSTER_ID" ]]; then
 fi
 
 if [[ -n "$CREDENTIALS" ]]; then
-  EXPORTER_ARGS+=(-u "$CREDENTIALS")
+  EXPORTER_ARGS+=(--credentials "$CREDENTIALS")
 fi
 
 if [[ -n "$SSL_ENABLED" ]]; then
-  EXPORTER_ARGS+=(-ssl "$SSL_ENABLED")
+  EXPORTER_ARGS+=(--ssl-enabled "$SSL_ENABLED")
 fi
 
 exec "$JAVACMD" \
diff --git a/solr/prometheus-exporter/bin/solr-exporter.cmd b/solr/prometheus-exporter/bin/solr-exporter.cmd
index 7e9dd2a91f2..c55133db616 100644
--- a/solr/prometheus-exporter/bin/solr-exporter.cmd
+++ b/solr/prometheus-exporter/bin/solr-exporter.cmd
@@ -80,11 +80,11 @@ IF NOT "%CONFIG_FILE%"=="" set EXPORTER_ARGS=%EXPORTER_ARGS% --config-file %CONF
 IF NOT "%PORT%"=="" set EXPORTER_ARGS=%EXPORTER_ARGS% -p %PORT%
 IF NOT "%SCRAPE_INTERVAL%"=="" set EXPORTER_ARGS=%EXPORTER_ARGS% --scrape-interval %SCRAPE_INTERVAL%
 IF NOT "%NUM_THREADS%"=="" set EXPORTER_ARGS=%EXPORTER_ARGS% --num-threads %NUM_THREADS%
-IF NOT "%ZK_HOST%"=="" set EXPORTER_ARGS=%EXPORTER_ARGS% -z %ZK_HOST%
-IF NOT "%SOLR_URL%"=="" set EXPORTER_ARGS=%EXPORTER_ARGS% -b %SOLR_URL%
+IF NOT "%ZK_HOST%"=="" set EXPORTER_ARGS=%EXPORTER_ARGS% --zk-host %ZK_HOST%
+IF NOT "%SOLR_URL%"=="" set EXPORTER_ARGS=%EXPORTER_ARGS% --solr-url %SOLR_URL%
 IF NOT "%CLUSTER_ID%"=="" set EXPORTER_ARGS=%EXPORTER_ARGS% --cluster-id "%CLUSTER_ID%"
-IF NOT "%CREDENTIALS%"=="" set EXPORTER_ARGS=%EXPORTER_ARGS% -u "%CREDENTIALS%"
-IF NOT "%SSL_ENABLED%"=="" set EXPORTER_ARGS=%EXPORTER_ARGS% -ssl "%SSL_ENABLED%"
+IF NOT "%CREDENTIALS%"=="" set EXPORTER_ARGS=%EXPORTER_ARGS% --credentials "%CREDENTIALS%"
+IF NOT "%SSL_ENABLED%"=="" set EXPORTER_ARGS=%EXPORTER_ARGS% --ssl-enabled "%SSL_ENABLED%"
 goto endInit
 
 @REM Reaching here means variables are defined and arguments have been captured
diff --git a/solr/prometheus-exporter/src/java/org/apache/solr/prometheus/exporter/SolrExporter.java b/solr/prometheus-exporter/src/java/org/apache/solr/prometheus/exporter/SolrExporter.java
index 2a839b0c794..ef24cda66a2 100644
--- a/solr/prometheus-exporter/src/java/org/apache/solr/prometheus/exporter/SolrExporter.java
+++ b/solr/prometheus-exporter/src/java/org/apache/solr/prometheus/exporter/SolrExporter.java
@@ -26,7 +26,6 @@
 import org.apache.commons.cli.CommandLine;
 import org.apache.commons.cli.CommandLineParser;
 import org.apache.commons.cli.DefaultParser;
-import org.apache.commons.cli.DeprecatedAttributes;
 import org.apache.commons.cli.HelpFormatter;
 import org.apache.commons.cli.Option;
 import org.apache.commons.cli.Options;
@@ -35,7 +34,6 @@
 import org.apache.solr.common.util.ExecutorUtil;
 import org.apache.solr.common.util.IOUtils;
 import org.apache.solr.common.util.SolrNamedThreadFactory;
-import org.apache.solr.common.util.StrUtils;
 import org.apache.solr.prometheus.collector.MetricsCollectorFactory;
 import org.apache.solr.prometheus.collector.SchedulerMetricsCollector;
 import org.apache.solr.prometheus.scraper.SolrCloudScraper;
@@ -51,7 +49,6 @@ public class SolrExporter {
   private static final int DEFAULT_PORT = 8989;
   private static final String DEFAULT_BASE_URL = "http://localhost:8983/solr";
   private static final String DEFAULT_ZK_HOST = "";
-  private static final String DEFAULT_CLUSTER_ID = "";
   private static final String DEFAULT_CONFIG = "solr-exporter-config.xml";
   private static final int DEFAULT_SCRAPE_INTERVAL = 60;
   private static final Integer DEFAULT_NUM_THREADS = 1;
@@ -143,36 +140,17 @@ private SolrScraper createScraper(
 
   public static void main(String[] args) {
     Options mainOptions = new Options();
-    Options deprecatedOptions = new Options();
 
-    // Change to -s and --solr-url in main once deprecated -s flag for --scrape-interval is removed.
-    Option baseUrlOption =
-        Option.builder("b")
-            .longOpt("base-url")
-            .hasArg()
-            .argName("BASE_URL")
-            .type(String.class)
-            .desc(
-                "Specify the Solr base URL when connecting to Solr in standalone mode. If omitted both the -b parameter and the -z parameter, connect to http://localhost:8983/solr. For example 'http://localhost:8983/solr'.")
-            .build();
-    mainOptions.addOption(baseUrlOption);
-
-    Option baseUrlDepOption =
-        Option.builder()
-            .longOpt("baseUrl")
+    Option solrUrlOption =
+        Option.builder("s")
+            .longOpt("solr-url")
             .hasArg()
             .argName("BASE_URL")
             .type(String.class)
-            .deprecated(
-                DeprecatedAttributes.builder()
-                    .setForRemoval(true)
-                    .setSince("9.7")
-                    .setDescription("Use --base-url instead")
-                    .get())
             .desc(
                 "Specify the Solr base URL when connecting to Solr in standalone mode. If omitted both the -b parameter and the -z parameter, connect to http://localhost:8983/solr. For example 'http://localhost:8983/solr'.")
             .build();
-    deprecatedOptions.addOption(baseUrlDepOption);
+    mainOptions.addOption(solrUrlOption);
 
     Option configOption =
         Option.builder()
@@ -182,21 +160,8 @@ public static void main(String[] args) {
             .type(String.class)
             .desc("Specify the configuration file; the default is " + DEFAULT_CONFIG + ".")
             .build();
-    Option configOptionDeprecated =
-        Option.builder("f")
-            .hasArg()
-            .argName("CONFIG")
-            .type(String.class)
-            .deprecated(
-                DeprecatedAttributes.builder()
-                    .setForRemoval(true)
-                    .setSince("9.8")
-                    .setDescription("Use --config-file instead")
-                    .get())
-            .desc("Specify the configuration file; the default is " + DEFAULT_CONFIG + ".")
-            .build();
+
     mainOptions.addOption(configOption);
-    mainOptions.addOption(configOptionDeprecated);
 
     Option helpOption =
         Option.builder("h").longOpt("help").desc("Prints this help message.").build();
@@ -213,22 +178,6 @@ public static void main(String[] args) {
             .build();
     mainOptions.addOption(clusterIdOption);
 
-    Option clusterIdDepOption =
-        Option.builder("i")
-            .deprecated(
-                DeprecatedAttributes.builder()
-                    .setForRemoval(true)
-                    .setSince("9.8")
-                    .setDescription("Use --cluster-id instead")
-                    .get())
-            .hasArg()
-            .argName("CLUSTER_ID")
-            .type(String.class)
-            .desc(
-                "Specify a unique identifier for the cluster, which can be used to select between multiple clusters in Grafana. By default this ID will be equal to a hash of the -b or -z argument")
-            .build();
-    deprecatedOptions.addOption(clusterIdDepOption);
-
     Option numThreadsOption =
         Option.builder()
             .longOpt("num-threads")
@@ -241,23 +190,6 @@ public static void main(String[] args) {
                     + ".")
             .build();
     mainOptions.addOption(numThreadsOption);
-    Option numThreadsOptionDeprecated =
-        Option.builder("n")
-            .hasArg()
-            .deprecated(
-                DeprecatedAttributes.builder()
-                    .setForRemoval(true)
-                    .setSince("9.8")
-                    .setDescription("Use --num-threads instead")
-                    .get())
-            .argName("NUM_THREADS")
-            .type(Integer.class)
-            .desc(
-                "Specify the number of threads. solr-exporter creates a thread pools for request to Solr. If you need to improve request latency via solr-exporter, you can increase the number of threads; the default is "
-                    + DEFAULT_NUM_THREADS
-                    + ".")
-            .build();
-    mainOptions.addOption(numThreadsOptionDeprecated);
 
     Option portOption =
         Option.builder("p")
@@ -282,26 +214,8 @@ public static void main(String[] args) {
             .build();
     mainOptions.addOption(scrapeIntervalOption);
 
-    Option scrapeIntervalOptionDeprecated =
-        Option.builder("s")
-            .hasArg()
-            .argName("SCRAPE_INTERVAL")
-            .type(Integer.class)
-            .deprecated(
-                DeprecatedAttributes.builder()
-                    .setForRemoval(true)
-                    .setSince("9.8")
-                    .setDescription("Use --scrape-interval instead")
-                    .get())
-            .desc(
-                "Specify the delay between scraping Solr metrics; the default is "
-                    + DEFAULT_SCRAPE_INTERVAL
-                    + " seconds.")
-            .build();
-    mainOptions.addOption(scrapeIntervalOptionDeprecated);
-
     Option sslOption =
-        Option.builder("ssl")
+        Option.builder()
             .longOpt("ssl-enabled")
             .type(Boolean.class)
             .desc(
@@ -333,7 +247,6 @@ public static void main(String[] args) {
 
     Options options = new Options();
     options.addOptions(mainOptions);
-    options.addOptions(deprecatedOptions);
 
     try {
       CommandLineParser parser = new DefaultParser();
@@ -353,12 +266,8 @@ public static void main(String[] args) {
         String zkHost = commandLine.getOptionValue(zkHostOption, DEFAULT_ZK_HOST);
         defaultClusterId = makeShortHash(zkHost);
         scrapeConfiguration = SolrScrapeConfiguration.solrCloud(zkHost);
-      } else if (commandLine.hasOption(baseUrlOption) || commandLine.hasOption(baseUrlDepOption)) {
-        log.warn("-b and --base-url will be replaced with -s and --solr-url in Solr 10");
-        String baseUrl =
-            commandLine.hasOption(baseUrlOption)
-                ? commandLine.getOptionValue(baseUrlOption)
-                : commandLine.getOptionValue(baseUrlDepOption, DEFAULT_BASE_URL);
+      } else if (commandLine.hasOption(solrUrlOption)) {
+        String baseUrl = commandLine.getOptionValue(solrUrlOption);
         defaultClusterId = makeShortHash(baseUrl);
         scrapeConfiguration = SolrScrapeConfiguration.standalone(baseUrl);
       }
@@ -366,18 +275,12 @@ public static void main(String[] args) {
       if (scrapeConfiguration == null) {
         log.error(
             "Must provide either --{} or --{}",
-            baseUrlOption.getLongOpt(),
+            solrUrlOption.getLongOpt(),
             zkHostOption.getLongOpt());
       }
 
       int port = commandLine.getParsedOptionValue(portOption, DEFAULT_PORT);
-      String clusterId = commandLine.getOptionValue(clusterIdOption, DEFAULT_CLUSTER_ID);
-      if (commandLine.hasOption("i")) {
-        clusterId = commandLine.getOptionValue("i");
-      }
-      if (StrUtils.isNullOrEmpty(clusterId)) {
-        clusterId = defaultClusterId;
-      }
+      String clusterId = commandLine.getOptionValue(clusterIdOption, defaultClusterId);
 
       if (commandLine.hasOption(credentialsOption)) {
         String credentials = commandLine.getOptionValue(credentialsOption, DEFAULT_CREDENTIALS);
@@ -397,25 +300,10 @@ public static void main(String[] args) {
             getSystemVariable("SOLR_SSL_TRUST_STORE_PASSWORD"));
       }
 
-      String configFile = DEFAULT_CONFIG;
-      if (commandLine.hasOption(configOptionDeprecated)) {
-        configFile = commandLine.getOptionValue(configOptionDeprecated);
-      } else if (commandLine.hasOption(configOption)) {
-        configFile = commandLine.getOptionValue(configOption);
-      }
-      int numberOfThreads = DEFAULT_NUM_THREADS;
-      if (commandLine.hasOption("num-threads")) {
-        numberOfThreads = commandLine.getParsedOptionValue("num-threads");
-      } else if (commandLine.hasOption("n")) {
-        numberOfThreads = commandLine.getParsedOptionValue("n");
-      }
-
-      int scrapeInterval = DEFAULT_SCRAPE_INTERVAL;
-      if (commandLine.hasOption("s")) {
-        scrapeInterval = commandLine.getParsedOptionValue(scrapeIntervalOptionDeprecated);
-      } else if (commandLine.hasOption("scrape-interval")) {
-        scrapeInterval = commandLine.getParsedOptionValue(scrapeIntervalOption);
-      }
+      String configFile = commandLine.getOptionValue(configOption, DEFAULT_CONFIG);
+      int numberOfThreads = commandLine.getParsedOptionValue("num-threads", DEFAULT_NUM_THREADS);
+      int scrapeInterval =
+          commandLine.getParsedOptionValue(scrapeIntervalOption, DEFAULT_SCRAPE_INTERVAL);
 
       SolrExporter solrExporter =
           new SolrExporter(
diff --git a/solr/solr-ref-guide/modules/deployment-guide/pages/enabling-ssl.adoc b/solr/solr-ref-guide/modules/deployment-guide/pages/enabling-ssl.adoc
index 34d540975a6..24704e8037b 100644
--- a/solr/solr-ref-guide/modules/deployment-guide/pages/enabling-ssl.adoc
+++ b/solr/solr-ref-guide/modules/deployment-guide/pages/enabling-ssl.adoc
@@ -236,7 +236,7 @@ The examples below use the `bin/solr` tool that comes with Solr to do this.
 ====
 [source,terminal]
 ----
-$ bin/solr cluster --property urlSchema --value https -z server1:2181,server2:2181,server3:2181
+$ bin/solr cluster --property urlSchema --value https --zk-host server1:2181,server2:2181,server3:2181
 ----
 ====
 
@@ -245,12 +245,12 @@ Windows::
 ====
 [source,powershell]
 ----
-C:\> bin/solr.cmd --property urlSchema --value https -z server1:2181,server2:2181,server3:2181
+C:\> bin/solr.cmd --property urlSchema --value https --zk-host server1:2181,server2:2181,server3:2181
 ----
 ====
 ======
 
-Be sure to use the correct `zkhost` value for your system.
+Be sure to use the correct `--zk-host` value for your system.
 If you have set up your ZooKeeper ensemble to use a xref:taking-solr-to-production.adoc#zookeeper-chroot[chroot for Solr], make sure to include it in the `zkHost` string, e.g., `--zk-host server1:2181,server2:2181,server3:2181/solr`.
 
 === Update Cluster Properties for Existing Collections
diff --git a/solr/solr-ref-guide/modules/deployment-guide/pages/monitoring-with-prometheus-and-grafana.adoc b/solr/solr-ref-guide/modules/deployment-guide/pages/monitoring-with-prometheus-and-grafana.adoc
index 898be24b339..006888ede88 100644
--- a/solr/solr-ref-guide/modules/deployment-guide/pages/monitoring-with-prometheus-and-grafana.adoc
+++ b/solr/solr-ref-guide/modules/deployment-guide/pages/monitoring-with-prometheus-and-grafana.adoc
@@ -62,7 +62,7 @@ Linux::
 [source,bash]
 ----
 $ cd prometheus-exporter
-$ ./bin/solr-exporter -p 9854 -b http://localhost:8983/solr --config-file ./conf/solr-exporter-config.xml --num-threads 8
+$ ./bin/solr-exporter -p 9854 --solr-url http://localhost:8983/solr --config-file ./conf/solr-exporter-config.xml --num-threads 8
 ----
 
 .SolrCloud
@@ -80,7 +80,7 @@ Windows::
 [source,text]
 ----
 > cd prometheus-exporter
-> .\bin\solr-exporter.cmd -p 9854 -b http://localhost:8983/solr --config-file .\conf\solr-exporter-config.xml --num-threads 8
+> .\bin\solr-exporter.cmd -p 9854 --solr-url http://localhost:8983/solr --config-file .\conf\solr-exporter-config.xml --num-threads 8
 ----
 
 .SolrCloud
@@ -117,16 +117,16 @@ The port where Prometheus will listen for new data.
 This port will be used to configure Prometheus.
 It can be any port not already in use on your server.
 
-`-b`, `--baseurl`, `$SOLR_URL`::
+`-s`, `--solr-url`, `$SOLR_URL`::
 +
 [%autowidth,frame=none]
 |===
 |Optional |Default: _see description_
 |===
 +
-The Solr base URL (such as `\http://localhost:8983/solr`) when Solr is running in a  The Solr base URL (such as `\http://localhost:8983/solr`) when Solr is running in a user-managed cluster or a single-node installation.
+The Solr base URL (such as `http://localhost:8983/solr`) when Solr is running in a user-managed cluster or a single-node installation.
 If you are running SolrCloud, do not specify this parameter.
-If neither the `-b` parameter nor the `-z` parameter are defined, the default is `-b \http://localhost:8983/solr`.
+If neither the `-s` parameter nor the `-z` parameter are defined, the default is `--solr-url http://localhost:8983/solr`.
 
 `-z`, `--zk-host`, `$ZK_HOST`::
 +
@@ -137,7 +137,7 @@ If neither the `-b` parameter nor the `-z` parameter are defined, the default is
 +
 The ZooKeeper connect string (such as `localhost:9983`, or `localhost:2181/solr`) when Solr is running SolrCloud.
 If you are running a user-managed cluster or single-node installation, do not specify this parameter.
-If neither the `-b` parameter nor the `-z` parameter are defined, the `-b` parameter default is used.
+If neither the `--solr-url` parameter nor the `-z` parameter are defined, the `--solr-url` parameter default is used.
 
 `--config-file`, `$CONFIG_FILE`::
 +
@@ -178,7 +178,7 @@ The freshness of the metrics can be improved by reducing the scrape interval but
 |Optional |Default: _see description_
 |===
 +
-A unique ID for the cluster to monitor. This ID will be added to all metrics as a label `cluster_id` and can be used as a filter in the Grafana dashboard if you operate multiple Solr clusters reporting to the same Prometheus instance. If this option is omitted, a hash of the `baseUrl` or `zkHost` will be used as ID by default.
+A unique ID for the cluster to monitor. This ID will be added to all metrics as a label `cluster_id` and can be used as a filter in the Grafana dashboard if you operate multiple Solr clusters reporting to the same Prometheus instance. If this option is omitted, a hash of the `--solr-url` or `--zk-host` will be used as ID by default.
 
 `-u`, `--credentials`, `$CREDENTIALS`::
 +
@@ -190,7 +190,7 @@ A unique ID for the cluster to monitor. This ID will be added to all metrics as
 Specify the credentials in the format `username:password`. Example: `--credentials solr:SolrRocks`.
 
 
-`-ssl`, `--ssl-enabled`, `$SSL_ENABLED`::
+`--ssl-enabled`, `$SSL_ENABLED`::
 +
 [%autowidth,frame=none]
 |===

From 0f8decddaa1a9b84e98b995b2f631dd9ac9684cd Mon Sep 17 00:00:00 2001
From: Houston Putman <houston@apache.org>
Date: Fri, 1 Nov 2024 16:44:17 -0500
Subject: [PATCH 17/32] BasicDistributedZk2Test - wait for ZK connection to
 come back

---
 .../solr/cloud/AbstractBasicDistributedZk2TestBase.java   | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/solr/test-framework/src/java/org/apache/solr/cloud/AbstractBasicDistributedZk2TestBase.java b/solr/test-framework/src/java/org/apache/solr/cloud/AbstractBasicDistributedZk2TestBase.java
index 056c5e69c5e..513596533ff 100644
--- a/solr/test-framework/src/java/org/apache/solr/cloud/AbstractBasicDistributedZk2TestBase.java
+++ b/solr/test-framework/src/java/org/apache/solr/cloud/AbstractBasicDistributedZk2TestBase.java
@@ -149,6 +149,14 @@ public void test() throws Exception {
       // expire a session...
       CloudJettyRunner cloudJetty = shardToJetty.get(SHARD1).get(0);
       chaosMonkey.expireSession(cloudJetty.jetty);
+      // Wait until the jetty is reconnected, otherwise the following index command could fail
+      cloudJetty
+          .jetty
+          .getCoreContainer()
+          .getZkController()
+          .getZkClient()
+          .getCuratorFramework()
+          .blockUntilConnected(50, TimeUnit.MILLISECONDS);
 
       indexr("id", docId + 1, t1, "slip this doc in");
 

From 4d482b2ad4a1815cffd4c9d38f7000e7a5c994c7 Mon Sep 17 00:00:00 2001
From: Houston Putman <houston@apache.org>
Date: Fri, 1 Nov 2024 17:34:22 -0500
Subject: [PATCH 18/32] SOLR-16116: Catch IllegalStateException in
 OverseerTaskProcessor

---
 .../src/java/org/apache/solr/cloud/OverseerTaskProcessor.java   | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/solr/core/src/java/org/apache/solr/cloud/OverseerTaskProcessor.java b/solr/core/src/java/org/apache/solr/cloud/OverseerTaskProcessor.java
index b63afbcf0c4..b56ed04f0f9 100644
--- a/solr/core/src/java/org/apache/solr/cloud/OverseerTaskProcessor.java
+++ b/solr/core/src/java/org/apache/solr/cloud/OverseerTaskProcessor.java
@@ -497,6 +497,8 @@ protected LeaderStatus amILeader() {
       } else {
         log.debug("", e);
       }
+    } catch (IllegalStateException e) {
+      success = false;
     } catch (InterruptedException e) {
       success = false;
       Thread.currentThread().interrupt();

From 206904ef3c446cc290daacd2f77ea553bbdd33ad Mon Sep 17 00:00:00 2001
From: Matthew Biscocho <54160956+mlbiscoc@users.noreply.github.com>
Date: Fri, 1 Nov 2024 23:30:11 -0400
Subject: [PATCH 19/32] SOLR-17530: Metrics: fix Prometheus response writer for
 TLOG & PULL replicas (#2795)

The new Prometheus response writer wasn't detecting TLOG or PULL replicas properly.

---------

Co-authored-by: David Smiley <dsmiley@apache.org>
---
 solr/CHANGES.txt                              |  6 ++++
 .../core/PrometheusCoreFormatterInfo.java     |  2 +-
 .../metrics/SolrPrometheusFormatterTest.java  | 31 +++++++++++++++++++
 3 files changed, 38 insertions(+), 1 deletion(-)

diff --git a/solr/CHANGES.txt b/solr/CHANGES.txt
index 8ed458efdb0..a5d34104a1a 100644
--- a/solr/CHANGES.txt
+++ b/solr/CHANGES.txt
@@ -217,6 +217,12 @@ led to the suppression of exceptions. (Andrey Bozhko)
 
 * SOLR-11318: Introduce unit testing for AssertTool. (Eric Pugh, Jason Gerlowski)
 
+==================  9.7.1 ==================
+Bug Fixes
+---------------------
+* SOLR-17530: Metrics: Thew new Prometheus response writer wasn't detecting TLOG or PULL replicas properly.
+  (Matthew Biscocho)
+
 ==================  9.7.0 ==================
 New Features
 ---------------------
diff --git a/solr/core/src/java/org/apache/solr/metrics/prometheus/core/PrometheusCoreFormatterInfo.java b/solr/core/src/java/org/apache/solr/metrics/prometheus/core/PrometheusCoreFormatterInfo.java
index 51f5d9765d9..baa7be5898b 100644
--- a/solr/core/src/java/org/apache/solr/metrics/prometheus/core/PrometheusCoreFormatterInfo.java
+++ b/solr/core/src/java/org/apache/solr/metrics/prometheus/core/PrometheusCoreFormatterInfo.java
@@ -33,5 +33,5 @@ enum CoreCategory {
     CORE
   }
 
-  Pattern CLOUD_CORE_PATTERN = Pattern.compile("^core_(.*)_(shard[0-9]+)_(replica_n[0-9]+)$");
+  Pattern CLOUD_CORE_PATTERN = Pattern.compile("^core_(.*)_(shard[0-9]+)_(replica_.[0-9]+)$");
 }
diff --git a/solr/core/src/test/org/apache/solr/metrics/SolrPrometheusFormatterTest.java b/solr/core/src/test/org/apache/solr/metrics/SolrPrometheusFormatterTest.java
index 186c83c8a91..5c62910b43f 100644
--- a/solr/core/src/test/org/apache/solr/metrics/SolrPrometheusFormatterTest.java
+++ b/solr/core/src/test/org/apache/solr/metrics/SolrPrometheusFormatterTest.java
@@ -16,6 +16,8 @@
  */
 package org.apache.solr.metrics;
 
+import static org.apache.solr.metrics.prometheus.core.PrometheusCoreFormatterInfo.CLOUD_CORE_PATTERN;
+
 import com.codahale.metrics.Counter;
 import com.codahale.metrics.Gauge;
 import com.codahale.metrics.Meter;
@@ -29,6 +31,7 @@
 import java.util.List;
 import java.util.Map;
 import java.util.concurrent.TimeUnit;
+import java.util.regex.Matcher;
 import org.apache.solr.SolrTestCaseJ4;
 import org.apache.solr.metrics.prometheus.SolrMetric;
 import org.apache.solr.metrics.prometheus.SolrPrometheusFormatter;
@@ -141,6 +144,34 @@ public Map<String, Number> getValue() {
     assertEquals(expectedLabels, actual.getLabels());
   }
 
+  @Test
+  public void testCloudCorePattern() {
+    String coreName = "core_test-core_shard2_replica_t123";
+    Matcher m = CLOUD_CORE_PATTERN.matcher(coreName);
+    assertTrue(m.find());
+    assertEquals("test-core", m.group(1));
+    assertEquals("shard2", m.group(2));
+    assertEquals("replica_t123", m.group(3));
+
+    coreName = "core_foo_bar_shard24_replica_p8";
+    m = CLOUD_CORE_PATTERN.matcher(coreName);
+    assertTrue(m.matches());
+    assertEquals("foo_bar", m.group(1));
+    assertEquals("shard24", m.group(2));
+    assertEquals("replica_p8", m.group(3));
+  }
+
+  @Test
+  public void testBadCloudCorePattern() {
+    String badCoreName = "core_solrtest_shard100_replica_xyz23";
+    Matcher m = CLOUD_CORE_PATTERN.matcher(badCoreName);
+    assertFalse(m.matches());
+
+    badCoreName = "core_solrtest_shards100_replica_x23";
+    m = CLOUD_CORE_PATTERN.matcher(badCoreName);
+    assertFalse(m.matches());
+  }
+
   static class TestSolrPrometheusFormatter extends SolrPrometheusFormatter {
     @Override
     public void exportDropwizardMetric(Metric dropwizardMetric, String metricName) {}

From cd681c073c3c0201aa49a3f243bb284c9880f26c Mon Sep 17 00:00:00 2001
From: David Smiley <dsmiley@salesforce.com>
Date: Mon, 4 Nov 2024 13:03:05 -0500
Subject: [PATCH 20/32] SOLR-17534: Add ClusterState.getCollectionNames (#2826)

Refactoring to introduce getCollectionNames(). The motivation is to reduce callers of getCollectionsMap and getCollectionStates, which will go away soon.
---
 solr/CHANGES.txt                              |  2 ++
 .../java/org/apache/solr/cli/DeleteTool.java  |  3 ++-
 .../apache/solr/handler/admin/ColStatus.java  |  3 +--
 .../solr/cloud/ClusterStateMockUtilTest.java  |  6 +++---
 .../apache/solr/cloud/ClusterStateTest.java   |  6 +++---
 .../org/apache/solr/cloud/OverseerTest.java   |  2 +-
 .../solr/cloud/ReindexCollectionTest.java     |  2 +-
 .../apache/solr/handler/sql/SolrSchema.java   | 20 +++++--------------
 .../prometheus/scraper/SolrCloudScraper.java  |  5 +++--
 .../solr/client/solrj/io/sql/JdbcTest.java    |  5 +----
 .../solr/common/cloud/ZkStateReader.java      |  2 +-
 .../solr/common/cloud/ClusterState.java       | 11 ++++++++++
 .../cloud/AbstractFullDistribZkTestBase.java  |  2 +-
 .../solr/cloud/MiniSolrCloudCluster.java      |  4 ++--
 14 files changed, 37 insertions(+), 36 deletions(-)

diff --git a/solr/CHANGES.txt b/solr/CHANGES.txt
index a5d34104a1a..35070264402 100644
--- a/solr/CHANGES.txt
+++ b/solr/CHANGES.txt
@@ -217,6 +217,8 @@ led to the suppression of exceptions. (Andrey Bozhko)
 
 * SOLR-11318: Introduce unit testing for AssertTool. (Eric Pugh, Jason Gerlowski)
 
+* SOLR-17534: Introduce ClusterState.getCollectionNames, a convenience method (David Smiley)
+
 ==================  9.7.1 ==================
 Bug Fixes
 ---------------------
diff --git a/solr/core/src/java/org/apache/solr/cli/DeleteTool.java b/solr/core/src/java/org/apache/solr/cli/DeleteTool.java
index 069128663b4..bd6d5f97cc2 100644
--- a/solr/core/src/java/org/apache/solr/cli/DeleteTool.java
+++ b/solr/core/src/java/org/apache/solr/cli/DeleteTool.java
@@ -20,6 +20,7 @@
 
 import java.io.PrintStream;
 import java.lang.invoke.MethodHandles;
+import java.util.Collection;
 import java.util.List;
 import java.util.Locale;
 import java.util.Optional;
@@ -155,7 +156,7 @@ protected void deleteCollection(CloudSolrClient cloudSolrClient, CommandLine cli
             configName);
       } else {
         // need to scan all Collections to see if any are using the config
-        Set<String> collections = zkStateReader.getClusterState().getCollectionsMap().keySet();
+        Collection<String> collections = zkStateReader.getClusterState().getCollectionNames();
 
         // give a little note to the user if there are many collections in case it takes a while
         if (collections.size() > 50)
diff --git a/solr/core/src/java/org/apache/solr/handler/admin/ColStatus.java b/solr/core/src/java/org/apache/solr/handler/admin/ColStatus.java
index 5bc0884a144..e7fe44e0fe6 100644
--- a/solr/core/src/java/org/apache/solr/handler/admin/ColStatus.java
+++ b/solr/core/src/java/org/apache/solr/handler/admin/ColStatus.java
@@ -21,7 +21,6 @@
 import java.lang.invoke.MethodHandles;
 import java.util.Collection;
 import java.util.Collections;
-import java.util.HashSet;
 import java.util.Map;
 import java.util.Set;
 import java.util.TreeMap;
@@ -76,7 +75,7 @@ public void getColStatus(NamedList<Object> results) {
     Collection<String> collections;
     String col = props.getStr(ZkStateReader.COLLECTION_PROP);
     if (col == null) {
-      collections = new HashSet<>(clusterState.getCollectionStates().keySet());
+      collections = clusterState.getCollectionNames();
     } else {
       collections = Collections.singleton(col);
     }
diff --git a/solr/core/src/test/org/apache/solr/cloud/ClusterStateMockUtilTest.java b/solr/core/src/test/org/apache/solr/cloud/ClusterStateMockUtilTest.java
index 3b4cc30ab4d..72707e70f84 100644
--- a/solr/core/src/test/org/apache/solr/cloud/ClusterStateMockUtilTest.java
+++ b/solr/core/src/test/org/apache/solr/cloud/ClusterStateMockUtilTest.java
@@ -36,7 +36,7 @@ public void testBuildClusterState_Simple() {
         ClusterStateMockUtil.buildClusterState("csr", "baseUrl1:8983_")) {
       ClusterState clusterState = zkStateReader.getClusterState();
       assertNotNull(clusterState);
-      assertEquals(1, clusterState.getCollectionStates().size());
+      assertEquals(1, clusterState.size());
       DocCollection collection1 = clusterState.getCollectionOrNull("collection1");
       assertNotNull(collection1);
       assertEquals(DocRouter.DEFAULT, collection1.getRouter());
@@ -62,7 +62,7 @@ public void testBuildClusterState_ReplicaTypes() {
         ClusterStateMockUtil.buildClusterState("csntp", "baseUrl1:8983_")) {
       ClusterState clusterState = zkStateReader.getClusterState();
       assertNotNull(clusterState);
-      assertEquals(1, clusterState.getCollectionStates().size());
+      assertEquals(1, clusterState.size());
       DocCollection collection1 = clusterState.getCollectionOrNull("collection1");
       assertNotNull(collection1);
       assertEquals(DocRouter.DEFAULT, collection1.getRouter());
@@ -83,7 +83,7 @@ public void testBuildClusterState_ReplicaStateAndType() {
         ClusterStateMockUtil.buildClusterState("csrStRpDnF", "baseUrl1:8983_")) {
       ClusterState clusterState = zkStateReader.getClusterState();
       assertNotNull(clusterState);
-      assertEquals(1, clusterState.getCollectionStates().size());
+      assertEquals(1, clusterState.size());
       DocCollection collection1 = clusterState.getCollectionOrNull("collection1");
       assertNotNull(collection1);
       assertEquals(DocRouter.DEFAULT, collection1.getRouter());
diff --git a/solr/core/src/test/org/apache/solr/cloud/ClusterStateTest.java b/solr/core/src/test/org/apache/solr/cloud/ClusterStateTest.java
index 068a8f38a4d..e88193b4a3c 100644
--- a/solr/core/src/test/org/apache/solr/cloud/ClusterStateTest.java
+++ b/solr/core/src/test/org/apache/solr/cloud/ClusterStateTest.java
@@ -81,7 +81,7 @@ public void testStoreAndRead() {
 
     assertEquals(
         "Provided liveNodes not used properly", 2, loadedClusterState.getLiveNodes().size());
-    assertEquals("No collections found", 2, loadedClusterState.getCollectionsMap().size());
+    assertEquals("No collections found", 2, loadedClusterState.size());
     assertEquals(
         "Properties not copied properly",
         replica.getStr("prop1"),
@@ -109,13 +109,13 @@ public void testStoreAndRead() {
 
     assertEquals(
         "Provided liveNodes not used properly", 2, loadedClusterState.getLiveNodes().size());
-    assertEquals("Should not have collections", 0, loadedClusterState.getCollectionsMap().size());
+    assertEquals("Should not have collections", 0, loadedClusterState.size());
 
     loadedClusterState =
         ClusterState.createFromJson(-1, (byte[]) null, liveNodes, Instant.now(), null);
 
     assertEquals(
         "Provided liveNodes not used properly", 2, loadedClusterState.getLiveNodes().size());
-    assertEquals("Should not have collections", 0, loadedClusterState.getCollectionsMap().size());
+    assertEquals("Should not have collections", 0, loadedClusterState.size());
   }
 }
diff --git a/solr/core/src/test/org/apache/solr/cloud/OverseerTest.java b/solr/core/src/test/org/apache/solr/cloud/OverseerTest.java
index 1350106457d..ae10c722b20 100644
--- a/solr/core/src/test/org/apache/solr/cloud/OverseerTest.java
+++ b/solr/core/src/test/org/apache/solr/cloud/OverseerTest.java
@@ -758,7 +758,7 @@ private void waitForCollections(ZkStateReader stateReader, String... collections
     while (0 < maxIterations--) {
 
       final ClusterState state = stateReader.getClusterState();
-      Set<String> availableCollections = state.getCollectionsMap().keySet();
+      Set<String> availableCollections = (Set<String>) state.getCollectionNames();
       int availableCount = 0;
       for (String requiredCollection : collections) {
         stateReader.waitForState(
diff --git a/solr/core/src/test/org/apache/solr/cloud/ReindexCollectionTest.java b/solr/core/src/test/org/apache/solr/cloud/ReindexCollectionTest.java
index 29e85f66905..200e7b68974 100644
--- a/solr/core/src/test/org/apache/solr/cloud/ReindexCollectionTest.java
+++ b/solr/core/src/test/org/apache/solr/cloud/ReindexCollectionTest.java
@@ -192,7 +192,7 @@ private void doTestSameTargetReindexing(boolean sourceRemove, boolean followAlia
     String prefix = ReindexCollectionCmd.TARGET_COL_PREFIX + targetCollection;
     while (!timeOut.hasTimedOut()) {
       timeOut.sleep(500);
-      for (String name : cloudManager.getClusterState().getCollectionsMap().keySet()) {
+      for (String name : cloudManager.getClusterState().getCollectionNames()) {
         if (name.startsWith(prefix)) {
           realTargetCollection = name;
           break;
diff --git a/solr/modules/sql/src/java/org/apache/solr/handler/sql/SolrSchema.java b/solr/modules/sql/src/java/org/apache/solr/handler/sql/SolrSchema.java
index 6d63e67c3dc..376c5df343e 100644
--- a/solr/modules/sql/src/java/org/apache/solr/handler/sql/SolrSchema.java
+++ b/solr/modules/sql/src/java/org/apache/solr/handler/sql/SolrSchema.java
@@ -18,6 +18,7 @@
 
 import java.io.Closeable;
 import java.io.IOException;
+import java.util.Collection;
 import java.util.Date;
 import java.util.EnumSet;
 import java.util.HashMap;
@@ -87,23 +88,12 @@ protected Map<String, Table> getTableMap() {
     String zk = this.properties.getProperty("zk");
     CloudSolrClient cloudSolrClient = solrClientCache.getCloudSolrClient(zk);
     ClusterState clusterState = cloudSolrClient.getClusterState();
-
-    final Map<String, Table> builder = new HashMap<>();
-
-    Set<String> collections = clusterState.getCollectionsMap().keySet();
-    for (String collection : collections) {
-      builder.put(collection, new SolrTable(this, collection));
-    }
-
     Aliases aliases = ZkStateReader.from(cloudSolrClient).getAliases();
-    for (String alias : aliases.getCollectionAliasListMap().keySet()) {
-      // don't create duplicate entries
-      if (!collections.contains(alias)) {
-        builder.put(alias, new SolrTable(this, alias));
-      }
-    }
 
-    return Map.copyOf(builder);
+    Collection<String> collectionNames = clusterState.getCollectionNames();
+    Set<String> aliasNames = aliases.getCollectionAliasListMap().keySet();
+    return Stream.concat(collectionNames.stream(), aliasNames.stream())
+        .collect(Collectors.toMap(n -> n, n -> new SolrTable(this, n), (t1, t2) -> t1));
   }
 
   private Map<String, LukeResponse.FieldInfo> getFieldInfo(final String collection) {
diff --git a/solr/prometheus-exporter/src/java/org/apache/solr/prometheus/scraper/SolrCloudScraper.java b/solr/prometheus-exporter/src/java/org/apache/solr/prometheus/scraper/SolrCloudScraper.java
index 55ded335c1d..e333076dbe2 100644
--- a/solr/prometheus-exporter/src/java/org/apache/solr/prometheus/scraper/SolrCloudScraper.java
+++ b/solr/prometheus-exporter/src/java/org/apache/solr/prometheus/scraper/SolrCloudScraper.java
@@ -19,6 +19,7 @@
 import com.github.benmanes.caffeine.cache.Cache;
 import com.github.benmanes.caffeine.cache.Caffeine;
 import java.io.IOException;
+import java.util.Collection;
 import java.util.List;
 import java.util.Map;
 import java.util.Set;
@@ -137,8 +138,8 @@ private Set<String> getBaseUrls() throws IOException {
         .collect(Collectors.toSet());
   }
 
-  private Set<String> getCollections() throws IOException {
-    return solrClient.getClusterState().getCollectionStates().keySet();
+  private Collection<String> getCollections() throws IOException {
+    return solrClient.getClusterState().getCollectionNames();
   }
 
   @Override
diff --git a/solr/solrj-streaming/src/test/org/apache/solr/client/solrj/io/sql/JdbcTest.java b/solr/solrj-streaming/src/test/org/apache/solr/client/solrj/io/sql/JdbcTest.java
index 822f7d38350..12b9f807847 100644
--- a/solr/solrj-streaming/src/test/org/apache/solr/client/solrj/io/sql/JdbcTest.java
+++ b/solr/solrj-streaming/src/test/org/apache/solr/client/solrj/io/sql/JdbcTest.java
@@ -29,8 +29,6 @@
 import java.util.Arrays;
 import java.util.List;
 import java.util.Properties;
-import java.util.Set;
-import java.util.SortedSet;
 import java.util.TreeSet;
 import org.apache.lucene.tests.util.LuceneTestCase;
 import org.apache.solr.client.solrj.impl.CloudSolrClient;
@@ -652,8 +650,7 @@ private void testJDBCMethods(
       solrClient.connect();
       ZkStateReader zkStateReader = ZkStateReader.from(solrClient);
 
-      Set<String> collectionsSet = zkStateReader.getClusterState().getCollectionsMap().keySet();
-      SortedSet<String> tables = new TreeSet<>(collectionsSet);
+      var tables = new TreeSet<String>(zkStateReader.getClusterState().getCollectionNames());
 
       Aliases aliases = zkStateReader.getAliases();
       tables.addAll(aliases.getCollectionAliasListMap().keySet());
diff --git a/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/ZkStateReader.java b/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/ZkStateReader.java
index df638085a2e..be5c4249d0a 100644
--- a/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/ZkStateReader.java
+++ b/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/ZkStateReader.java
@@ -625,7 +625,7 @@ private void constructState(Set<String> changedCollections) {
           collectionWatches.watchedCollections().size(),
           collectionWatches.activeCollectionCount(),
           lazyCollectionStates.keySet().size(),
-          clusterState.getCollectionStates().size());
+          clusterState.size());
     }
 
     if (log.isTraceEnabled()) {
diff --git a/solr/solrj/src/java/org/apache/solr/common/cloud/ClusterState.java b/solr/solrj/src/java/org/apache/solr/common/cloud/ClusterState.java
index b1d8782cb10..b1a42644a9d 100644
--- a/solr/solrj/src/java/org/apache/solr/common/cloud/ClusterState.java
+++ b/solr/solrj/src/java/org/apache/solr/common/cloud/ClusterState.java
@@ -154,6 +154,16 @@ public DocCollection getCollectionOrNull(String collectionName, boolean allowCac
     return ref == null ? null : ref.get(allowCached);
   }
 
+  /**
+   * The collection names. Like a Set but might not implement it. Immutable; non-null. Some names
+   * returned might not resolve via {@link #getCollectionOrNull(String)}, so consider this a close
+   * approximation.
+   */
+  public Collection<String> getCollectionNames() {
+    // should we document we are sorted too?  Albeit that ties our hands.
+    return immutableCollectionStates.keySet();
+  }
+
   /**
    * Get a map of collection name vs DocCollection objects
    *
@@ -471,6 +481,7 @@ public String toString() {
     }
   }
 
+  /** The approximate number of collections. */
   public int size() {
     return collectionStates.size();
   }
diff --git a/solr/test-framework/src/java/org/apache/solr/cloud/AbstractFullDistribZkTestBase.java b/solr/test-framework/src/java/org/apache/solr/cloud/AbstractFullDistribZkTestBase.java
index 568d8b7e373..dcdab4e86d1 100644
--- a/solr/test-framework/src/java/org/apache/solr/cloud/AbstractFullDistribZkTestBase.java
+++ b/solr/test-framework/src/java/org/apache/solr/cloud/AbstractFullDistribZkTestBase.java
@@ -1813,7 +1813,7 @@ protected void assertDocCounts(boolean verbose) throws Exception {
           "Could not find collection "
               + DEFAULT_COLLECTION
               + " in "
-              + clusterState.getCollectionsMap().keySet());
+              + clusterState.getCollectionNames());
     }
 
     for (CloudJettyRunner cjetty : cloudJettys) {
diff --git a/solr/test-framework/src/java/org/apache/solr/cloud/MiniSolrCloudCluster.java b/solr/test-framework/src/java/org/apache/solr/cloud/MiniSolrCloudCluster.java
index 54f23d71260..c600c7e4ffa 100644
--- a/solr/test-framework/src/java/org/apache/solr/cloud/MiniSolrCloudCluster.java
+++ b/solr/test-framework/src/java/org/apache/solr/cloud/MiniSolrCloudCluster.java
@@ -585,7 +585,7 @@ public void deleteAllCollections() throws Exception {
 
       reader.createClusterStateWatchersAndUpdate(); // up to date aliases & collections
       reader.aliasesManager.applyModificationAndExportToZk(aliases -> Aliases.EMPTY);
-      for (String collection : reader.getClusterState().getCollectionStates().keySet()) {
+      for (String collection : reader.getClusterState().getCollectionNames()) {
         CollectionAdminRequest.deleteCollection(collection).process(solrClient);
       }
 
@@ -595,7 +595,7 @@ public void deleteAllCollections() throws Exception {
             "Still waiting to see all collections removed from clusterstate.");
       }
 
-      for (String collection : reader.getClusterState().getCollectionStates().keySet()) {
+      for (String collection : reader.getClusterState().getCollectionNames()) {
         reader.waitForState(collection, 15, TimeUnit.SECONDS, Objects::isNull);
       }
     }

From f216c984d348c12cc4c4c24e24ee6bf014cc9b01 Mon Sep 17 00:00:00 2001
From: Solr Bot <125606113+solrbot@users.noreply.github.com>
Date: Mon, 4 Nov 2024 19:10:36 +0100
Subject: [PATCH 21/32] Update org.hamcrest:* to v3 (major) (#2617)

* Update org.hamcrest:* to v3

* Update hamcrest license file

---------

Co-authored-by: Christos Malliaridis <c.malliaridis@gmail.com>
---
 solr/licenses/hamcrest-2.2.jar.sha1      |  1 -
 solr/licenses/hamcrest-3.0.jar.sha1      |  1 +
 solr/licenses/hamcrest-LICENSE-BSD.txt   | 41 ++++++++++++------------
 solr/licenses/hamcrest-core-2.2.jar.sha1 |  1 -
 solr/licenses/hamcrest-core-3.0.jar.sha1 |  1 +
 versions.lock                            |  4 +--
 versions.props                           |  2 +-
 7 files changed, 26 insertions(+), 25 deletions(-)
 delete mode 100644 solr/licenses/hamcrest-2.2.jar.sha1
 create mode 100644 solr/licenses/hamcrest-3.0.jar.sha1
 delete mode 100644 solr/licenses/hamcrest-core-2.2.jar.sha1
 create mode 100644 solr/licenses/hamcrest-core-3.0.jar.sha1

diff --git a/solr/licenses/hamcrest-2.2.jar.sha1 b/solr/licenses/hamcrest-2.2.jar.sha1
deleted file mode 100644
index 820b1fb33c6..00000000000
--- a/solr/licenses/hamcrest-2.2.jar.sha1
+++ /dev/null
@@ -1 +0,0 @@
-1820c0968dba3a11a1b30669bb1f01978a91dedc
diff --git a/solr/licenses/hamcrest-3.0.jar.sha1 b/solr/licenses/hamcrest-3.0.jar.sha1
new file mode 100644
index 00000000000..6ccdd785a66
--- /dev/null
+++ b/solr/licenses/hamcrest-3.0.jar.sha1
@@ -0,0 +1 @@
+8fd9b78a8e6a6510a078a9e30e9e86a6035cfaf7
diff --git a/solr/licenses/hamcrest-LICENSE-BSD.txt b/solr/licenses/hamcrest-LICENSE-BSD.txt
index e3d4feb9cbb..c43f6b7b926 100644
--- a/solr/licenses/hamcrest-LICENSE-BSD.txt
+++ b/solr/licenses/hamcrest-LICENSE-BSD.txt
@@ -1,27 +1,28 @@
-BSD License
+BSD 3-Clause License
 
-Copyright (c) 2000-2006, www.hamcrest.org
-All rights reserved.
+Copyright (c) 2000-2024, www.hamcrest.org
 
 Redistribution and use in source and binary forms, with or without
 modification, are permitted provided that the following conditions are met:
 
-Redistributions of source code must retain the above copyright notice, this list of
-conditions and the following disclaimer. Redistributions in binary form must reproduce
-the above copyright notice, this list of conditions and the following disclaimer in
-the documentation and/or other materials provided with the distribution.
+1. Redistributions of source code must retain the above copyright notice, this
+   list of conditions and the following disclaimer.
 
-Neither the name of Hamcrest nor the names of its contributors may be used to endorse
-or promote products derived from this software without specific prior written
-permission.
+2. Redistributions in binary form must reproduce the above copyright notice,
+   this list of conditions and the following disclaimer in the documentation
+   and/or other materials provided with the distribution.
 
-THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY
-EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
-OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT
-SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
-INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED
-TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
-BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
-CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY
-WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH
-DAMAGE.
\ No newline at end of file
+3. Neither the name of the copyright holder nor the names of its
+   contributors may be used to endorse or promote products derived from
+   this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
diff --git a/solr/licenses/hamcrest-core-2.2.jar.sha1 b/solr/licenses/hamcrest-core-2.2.jar.sha1
deleted file mode 100644
index 8afb6048872..00000000000
--- a/solr/licenses/hamcrest-core-2.2.jar.sha1
+++ /dev/null
@@ -1 +0,0 @@
-3f2bd07716a31c395e2837254f37f21f0f0ab24b
diff --git a/solr/licenses/hamcrest-core-3.0.jar.sha1 b/solr/licenses/hamcrest-core-3.0.jar.sha1
new file mode 100644
index 00000000000..0801ceea637
--- /dev/null
+++ b/solr/licenses/hamcrest-core-3.0.jar.sha1
@@ -0,0 +1 @@
+254f69f4b0ca22198acfc19fcdd5f96140431c3e
diff --git a/versions.lock b/versions.lock
index 7418e07bfd9..11c37b2acd7 100644
--- a/versions.lock
+++ b/versions.lock
@@ -343,8 +343,8 @@ org.glassfish.jersey.core:jersey-server:3.1.9 (2 constraints: 491efe41)
 org.glassfish.jersey.ext:jersey-entity-filtering:3.1.9 (1 constraints: c3152bd3)
 org.glassfish.jersey.inject:jersey-hk2:3.1.9 (1 constraints: 0f050636)
 org.glassfish.jersey.media:jersey-media-json-jackson:3.1.9 (1 constraints: 0f050636)
-org.hamcrest:hamcrest:2.2 (3 constraints: 7620ce25)
-org.hamcrest:hamcrest-core:2.2 (1 constraints: cc05fe3f)
+org.hamcrest:hamcrest:3.0 (3 constraints: 74208125)
+org.hamcrest:hamcrest-core:3.0 (1 constraints: cc05fe3f)
 org.immutables:value-annotations:2.10.1 (1 constraints: 3605303b)
 org.itadaki:bzip2:0.9.1 (2 constraints: bd0c4b2c)
 org.javassist:javassist:3.30.2-GA (1 constraints: 2811fff0)
diff --git a/versions.props b/versions.props
index 0433da0fc4f..b9bb03803c3 100644
--- a/versions.props
+++ b/versions.props
@@ -62,7 +62,7 @@ org.glassfish.hk2*:*=3.1.1
 org.glassfish.hk2.external:jakarta.inject=2.6.1
 org.glassfish.jersey*:*=3.1.9
 org.glassfish.jersey.containers:jersey-container-jetty-http=2.39.1
-org.hamcrest:*=2.2
+org.hamcrest:*=3.0
 org.hsqldb:hsqldb=2.7.2
 org.immutables:value-annotations=2.10.1
 org.jctools:jctools-core=4.0.5

From e849fd0540fb4f0e013a1f73e93c3e85a933ed83 Mon Sep 17 00:00:00 2001
From: David Smiley <dsmiley@salesforce.com>
Date: Mon, 4 Nov 2024 17:32:39 -0500
Subject: [PATCH 22/32] SOLR-17456: TransactionLog ctor integrity (#2762)

The TransactionLog constructor can't handle an existing file being present; it shouldn't be there.
Should throw an exception in this case, NOT log a warning which would leave the object in a partially constructed state.
This should happen in the first place, of course.  I see no evidence it has occurred.
---
 .../java/org/apache/solr/update/TransactionLog.java    | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/solr/core/src/java/org/apache/solr/update/TransactionLog.java b/solr/core/src/java/org/apache/solr/update/TransactionLog.java
index 3174324f859..f8c0d48731d 100644
--- a/solr/core/src/java/org/apache/solr/update/TransactionLog.java
+++ b/solr/core/src/java/org/apache/solr/update/TransactionLog.java
@@ -38,6 +38,7 @@
 import java.util.concurrent.atomic.AtomicInteger;
 import org.apache.lucene.util.BytesRef;
 import org.apache.solr.common.SolrException;
+import org.apache.solr.common.SolrException.ErrorCode;
 import org.apache.solr.common.SolrInputDocument;
 import org.apache.solr.common.util.CollectionUtil;
 import org.apache.solr.common.util.DataInputInputStream;
@@ -221,8 +222,9 @@ protected TransactionLog(
         }
       } else {
         if (Files.exists(tlog)) {
-          log.warn("New transaction log already exists:{} size={}", tlog, Files.size(tlog));
-          return;
+          throw new SolrException(
+              ErrorCode.SERVER_ERROR,
+              "New transaction log already exists:" + tlog + " size=" + Files.size(tlog));
         }
 
         channel =
@@ -238,9 +240,6 @@ protected TransactionLog(
       }
 
       success = true;
-
-      assert ObjectReleaseTracker.track(this);
-
     } catch (IOException e) {
       throw new SolrException(SolrException.ErrorCode.SERVER_ERROR, e);
     } finally {
@@ -252,6 +251,7 @@ protected TransactionLog(
         }
       }
     }
+    assert ObjectReleaseTracker.track(this);
   }
 
   // for subclasses

From c5c538a9e025bda77ad591ee82beaa6a6732c408 Mon Sep 17 00:00:00 2001
From: David Smiley <dsmiley@apache.org>
Date: Mon, 4 Nov 2024 17:45:07 -0500
Subject: [PATCH 23/32] SOLR-17390: EmbeddedSolrServer now considers the
 ResponseParser (#2774)

And
* Moved HttpSolrCall.getResponseWriter to SolrQueryRequest
* Subtle improvements to make ContentStream work when they might not have
---
 solr/CHANGES.txt                              |   2 +
 .../solrj/embedded/EmbeddedSolrServer.java    | 162 +++++++++++-------
 .../apache/solr/request/SolrQueryRequest.java |  14 ++
 .../org/apache/solr/servlet/HttpSolrCall.java |   8 +-
 .../solr/servlet/SolrRequestParsers.java      |  11 +-
 .../solr/response/TestRawTransformer.java     |   7 +-
 .../apache/solr/client/solrj/SolrRequest.java |   3 +-
 .../client/solrj/impl/NoOpResponseParser.java |   3 +-
 8 files changed, 133 insertions(+), 77 deletions(-)

diff --git a/solr/CHANGES.txt b/solr/CHANGES.txt
index 35070264402..e8008ded0d1 100644
--- a/solr/CHANGES.txt
+++ b/solr/CHANGES.txt
@@ -154,6 +154,8 @@ Improvements
 
 * SOLR-17528: Introduce -y short option to bin/solr start --no-prompt option. Aligns with bin/solr package tool.  (Eric Pugh)
 
+* SOLR-17390: EmbeddedSolrServer now considers the ResponseParser (David Smiley)
+
 Optimizations
 ---------------------
 * SOLR-14985: Solrj CloudSolrClient with Solr URLs had serious performance regressions (since the
diff --git a/solr/core/src/java/org/apache/solr/client/solrj/embedded/EmbeddedSolrServer.java b/solr/core/src/java/org/apache/solr/client/solrj/embedded/EmbeddedSolrServer.java
index 49a887c9772..1da77d8db82 100644
--- a/solr/core/src/java/org/apache/solr/client/solrj/embedded/EmbeddedSolrServer.java
+++ b/solr/core/src/java/org/apache/solr/client/solrj/embedded/EmbeddedSolrServer.java
@@ -23,25 +23,28 @@
 import java.io.IOException;
 import java.io.InputStream;
 import java.nio.file.Path;
+import java.util.ArrayList;
 import java.util.Collection;
-import java.util.Collections;
-import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
 import java.util.Properties;
-import java.util.Set;
 import java.util.function.Supplier;
 import org.apache.lucene.search.TotalHits.Relation;
+import org.apache.solr.client.solrj.ResponseParser;
 import org.apache.solr.client.solrj.SolrClient;
 import org.apache.solr.client.solrj.SolrRequest;
 import org.apache.solr.client.solrj.SolrServerException;
 import org.apache.solr.client.solrj.StreamingResponseCallback;
 import org.apache.solr.client.solrj.impl.BinaryRequestWriter;
+import org.apache.solr.client.solrj.impl.BinaryResponseParser;
+import org.apache.solr.client.solrj.impl.InputStreamResponseParser;
 import org.apache.solr.client.solrj.request.ContentStreamUpdateRequest;
 import org.apache.solr.client.solrj.request.RequestWriter;
 import org.apache.solr.common.SolrDocument;
 import org.apache.solr.common.SolrDocumentList;
 import org.apache.solr.common.SolrException;
 import org.apache.solr.common.params.CommonParams;
-import org.apache.solr.common.params.ModifiableSolrParams;
+import org.apache.solr.common.params.MapSolrParams;
 import org.apache.solr.common.params.SolrParams;
 import org.apache.solr.common.util.ContentStream;
 import org.apache.solr.common.util.ContentStreamBase;
@@ -55,6 +58,7 @@
 import org.apache.solr.request.SolrRequestHandler;
 import org.apache.solr.request.SolrRequestInfo;
 import org.apache.solr.response.BinaryResponseWriter;
+import org.apache.solr.response.QueryResponseWriterUtil;
 import org.apache.solr.response.ResultContext;
 import org.apache.solr.response.SolrQueryResponse;
 import org.apache.solr.servlet.SolrRequestParsers;
@@ -165,13 +169,13 @@ public NamedList<Object> request(SolrRequest<?> request, String coreName)
       try {
         SolrQueryRequest req =
             _parser.buildRequestFrom(
-                null, request.getParams(), getContentStreams(request), request.getUserPrincipal());
+                null, getParams(request), getContentStreams(request), request.getUserPrincipal());
         req.getContext().put("httpMethod", request.getMethod().name());
         req.getContext().put(PATH, path);
         SolrQueryResponse resp = new SolrQueryResponse();
         handler.handleRequest(req, resp);
         checkForExceptions(resp);
-        return BinaryResponseWriter.getParsedResponse(req, resp);
+        return writeResponse(request, req, resp);
       } catch (IOException | SolrException iox) {
         throw iox;
       } catch (Exception ex) {
@@ -196,10 +200,7 @@ public NamedList<Object> request(SolrRequest<?> request, String coreName)
         throw new SolrException(SolrException.ErrorCode.SERVER_ERROR, "No such core: " + coreName);
       }
 
-      SolrParams params = request.getParams();
-      if (params == null) {
-        params = new ModifiableSolrParams();
-      }
+      SolrParams params = getParams(request);
 
       // Extract the handler from the path or params
       handler = core.getRequestHandler(path);
@@ -217,8 +218,10 @@ public NamedList<Object> request(SolrRequest<?> request, String coreName)
         throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "unknown handler: " + path);
       }
       req =
-          _parser.buildRequestFrom(
-              core, params, getContentStreams(request), request.getUserPrincipal());
+          core.getSolrConfig()
+              .getRequestParsers()
+              .buildRequestFrom(
+                  core, params, getContentStreams(request), request.getUserPrincipal());
       req.getContext().put(PATH, path);
       req.getContext().put("httpMethod", request.getMethod().name());
       SolrQueryResponse rsp = new SolrQueryResponse();
@@ -226,53 +229,7 @@ public NamedList<Object> request(SolrRequest<?> request, String coreName)
 
       core.execute(handler, req, rsp);
       checkForExceptions(rsp);
-
-      // Check if this should stream results
-      if (request.getStreamingResponseCallback() != null) {
-        try {
-          final StreamingResponseCallback callback = request.getStreamingResponseCallback();
-          BinaryResponseWriter.Resolver resolver =
-              new BinaryResponseWriter.Resolver(req, rsp.getReturnFields()) {
-                @Override
-                public void writeResults(ResultContext ctx, JavaBinCodec codec) throws IOException {
-                  // write an empty list...
-                  SolrDocumentList docs = new SolrDocumentList();
-                  docs.setNumFound(ctx.getDocList().matches());
-                  docs.setNumFoundExact(ctx.getDocList().hitCountRelation() == Relation.EQUAL_TO);
-                  docs.setStart(ctx.getDocList().offset());
-                  docs.setMaxScore(ctx.getDocList().maxScore());
-                  codec.writeSolrDocumentList(docs);
-
-                  // This will transform
-                  writeResultsBody(ctx, codec);
-                }
-              };
-
-          try (var out =
-              new ByteArrayOutputStream() {
-                ByteArrayInputStream toInputStream() {
-                  return new ByteArrayInputStream(buf, 0, count);
-                }
-              }) {
-            createJavaBinCodec(callback, resolver)
-                .setWritableDocFields(resolver)
-                .marshal(rsp.getValues(), out);
-
-            try (ByteArrayInputStream in = out.toInputStream()) {
-              @SuppressWarnings({"unchecked"})
-              NamedList<Object> resolved =
-                  (NamedList<Object>) new JavaBinCodec(resolver).unmarshal(in);
-              return resolved;
-            }
-          }
-        } catch (Exception ex) {
-          throw new RuntimeException(ex);
-        }
-      }
-
-      // Now write it out
-      NamedList<Object> normalized = BinaryResponseWriter.getParsedResponse(req, rsp);
-      return normalized;
+      return writeResponse(request, req, rsp);
     } catch (IOException | SolrException iox) {
       throw iox;
     } catch (Exception ex) {
@@ -285,12 +242,89 @@ ByteArrayInputStream toInputStream() {
     }
   }
 
-  private Set<ContentStream> getContentStreams(SolrRequest<?> request) throws IOException {
-    if (request.getMethod() == SolrRequest.METHOD.GET) return null;
+  private static SolrParams getParams(SolrRequest<?> request) {
+    var params = request.getParams();
+    var responseParser = request.getResponseParser();
+    if (responseParser == null) {
+      responseParser = new BinaryResponseParser();
+    }
+    var addParams =
+        new MapSolrParams(
+            Map.of(
+                CommonParams.WT,
+                responseParser.getWriterType(),
+                CommonParams.VERSION,
+                responseParser.getVersion()));
+    return SolrParams.wrapDefaults(addParams, params);
+  }
+
+  private NamedList<Object> writeResponse(
+      SolrRequest<?> request, SolrQueryRequest req, SolrQueryResponse rsp) throws IOException {
+    ResponseParser responseParser = request.getResponseParser();
+    if (responseParser == null) {
+      responseParser = new BinaryResponseParser();
+    }
+    StreamingResponseCallback callback = request.getStreamingResponseCallback();
+    // TODO refactor callback to be a special responseParser that we check for
+    // TODO if responseParser is a special/internal NamedList ResponseParser, just return NL
+
+    var byteBuffer =
+        new ByteArrayOutputStream() {
+          ByteArrayInputStream toInputStream() {
+            return new ByteArrayInputStream(buf, 0, count);
+          }
+        };
+
+    if (callback == null) {
+      QueryResponseWriterUtil.writeQueryResponse(
+          byteBuffer, req.getResponseWriter(), req, rsp, null);
+    } else {
+      // mostly stream results to the callback; rest goes into the byteBuffer
+      if (!(responseParser instanceof BinaryResponseParser))
+        throw new IllegalArgumentException(
+            "Only javabin is supported when using a streaming response callback");
+      var resolver =
+          new BinaryResponseWriter.Resolver(req, rsp.getReturnFields()) {
+            @Override
+            public void writeResults(ResultContext ctx, JavaBinCodec codec) throws IOException {
+              // write an empty list...
+              SolrDocumentList docs = new SolrDocumentList();
+              docs.setNumFound(ctx.getDocList().matches());
+              docs.setNumFoundExact(ctx.getDocList().hitCountRelation() == Relation.EQUAL_TO);
+              docs.setStart(ctx.getDocList().offset());
+              docs.setMaxScore(ctx.getDocList().maxScore());
+              codec.writeSolrDocumentList(docs);
+
+              // This will transform
+              writeResultsBody(ctx, codec);
+            }
+          };
+
+      // invoke callbacks, and writes the rest to byteBuffer
+      try (var javaBinCodec = createJavaBinCodec(callback, resolver)) {
+        javaBinCodec.setWritableDocFields(resolver).marshal(rsp.getValues(), byteBuffer);
+      }
+    }
+
+    if (responseParser instanceof InputStreamResponseParser) {
+      // SPECIAL CASE
+      NamedList<Object> namedList = new NamedList<>();
+      namedList.add("stream", byteBuffer.toInputStream());
+      namedList.add("responseStatus", 200); // always by this point
+      return namedList;
+    }
+
+    // note: don't bother using the Reader variant; it often throws UnsupportedOperationException
+    return responseParser.processResponse(byteBuffer.toInputStream(), null);
+  }
+
+  /** A list of streams, non-null. */
+  private List<ContentStream> getContentStreams(SolrRequest<?> request) throws IOException {
+    if (request.getMethod() == SolrRequest.METHOD.GET) return List.of();
     if (request instanceof ContentStreamUpdateRequest) {
       final ContentStreamUpdateRequest csur = (ContentStreamUpdateRequest) request;
       final Collection<ContentStream> cs = csur.getContentStreams();
-      if (cs != null) return new HashSet<>(cs);
+      if (cs != null) return new ArrayList<>(cs);
     }
 
     final RequestWriter.ContentWriter contentWriter = request.getContentWriter(null);
@@ -308,7 +342,7 @@ private Set<ContentStream> getContentStreams(SolrRequest<?> request) throws IOEx
 
     final byte[] buf = baos.toByteArray();
     if (buf.length > 0) {
-      return Collections.singleton(
+      return List.of(
           new ContentStreamBase() {
 
             @Override
@@ -323,7 +357,7 @@ public String getContentType() {
           });
     }
 
-    return null;
+    return List.of();
   }
 
   private JavaBinCodec createJavaBinCodec(
diff --git a/solr/core/src/java/org/apache/solr/request/SolrQueryRequest.java b/solr/core/src/java/org/apache/solr/request/SolrQueryRequest.java
index ab020ff2509..4b2a4d543d4 100644
--- a/solr/core/src/java/org/apache/solr/request/SolrQueryRequest.java
+++ b/solr/core/src/java/org/apache/solr/request/SolrQueryRequest.java
@@ -29,6 +29,7 @@
 import org.apache.solr.common.util.EnvUtils;
 import org.apache.solr.core.CoreContainer;
 import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.QueryResponseWriter;
 import org.apache.solr.schema.IndexSchema;
 import org.apache.solr.search.SolrIndexSearcher;
 import org.apache.solr.servlet.HttpSolrCall;
@@ -188,4 +189,17 @@ default CoreContainer getCoreContainer() {
   default CloudDescriptor getCloudDescriptor() {
     return getCore().getCoreDescriptor().getCloudDescriptor();
   }
+
+  /** The writer to use for this request, considering {@link CommonParams#WT}. Never null. */
+  default QueryResponseWriter getResponseWriter() {
+    // it's weird this method is here instead of SolrQueryResponse, but it's practical/convenient
+    SolrCore core = getCore();
+    String wt = getParams().get(CommonParams.WT);
+    if (core != null) {
+      return core.getQueryResponseWriter(wt);
+    } else {
+      return SolrCore.DEFAULT_RESPONSE_WRITERS.getOrDefault(
+          wt, SolrCore.DEFAULT_RESPONSE_WRITERS.get("standard"));
+    }
+  }
 }
diff --git a/solr/core/src/java/org/apache/solr/servlet/HttpSolrCall.java b/solr/core/src/java/org/apache/solr/servlet/HttpSolrCall.java
index 9a8fcd400b6..cc645476aef 100644
--- a/solr/core/src/java/org/apache/solr/servlet/HttpSolrCall.java
+++ b/solr/core/src/java/org/apache/solr/servlet/HttpSolrCall.java
@@ -934,13 +934,7 @@ protected void logAndFlushAdminRequest(SolrQueryResponse solrResp) throws IOExce
    * returns the default query response writer Note: This method must not return null
    */
   protected QueryResponseWriter getResponseWriter() {
-    String wt = solrReq.getParams().get(CommonParams.WT);
-    if (core != null) {
-      return core.getQueryResponseWriter(wt);
-    } else {
-      return SolrCore.DEFAULT_RESPONSE_WRITERS.getOrDefault(
-          wt, SolrCore.DEFAULT_RESPONSE_WRITERS.get("standard"));
-    }
+    return solrReq.getResponseWriter();
   }
 
   protected void handleAdmin(SolrQueryResponse solrResp) {
diff --git a/solr/core/src/java/org/apache/solr/servlet/SolrRequestParsers.java b/solr/core/src/java/org/apache/solr/servlet/SolrRequestParsers.java
index 38a45c71a91..ee459c29231 100644
--- a/solr/core/src/java/org/apache/solr/servlet/SolrRequestParsers.java
+++ b/solr/core/src/java/org/apache/solr/servlet/SolrRequestParsers.java
@@ -197,11 +197,18 @@ public SolrQueryRequest buildRequestFrom(
   private SolrQueryRequest buildRequestFrom(
       SolrCore core,
       SolrParams params,
-      Collection<ContentStream> streams,
+      Collection<ContentStream> streams, // might be added to but caller shouldn't depend on it
       RTimerTree requestTimer,
       final HttpServletRequest req,
       final Principal principal)
       throws Exception {
+    // ensure streams is non-null and mutable so we can easily add to it
+    if (streams == null) {
+      streams = new ArrayList<>();
+    } else if (!(streams instanceof ArrayList)) {
+      streams = new ArrayList<>(streams);
+    }
+
     // The content type will be applied to all streaming content
     String contentType = params.get(CommonParams.STREAM_CONTENTTYPE);
 
@@ -293,7 +300,7 @@ public HttpSolrCall getHttpSolrCall() {
             return httpSolrCall;
           }
         };
-    if (streams != null && streams.size() > 0) {
+    if (!streams.isEmpty()) {
       q.setContentStreams(streams);
     }
     return q;
diff --git a/solr/core/src/test/org/apache/solr/response/TestRawTransformer.java b/solr/core/src/test/org/apache/solr/response/TestRawTransformer.java
index b352f177e42..77de71eca38 100644
--- a/solr/core/src/test/org/apache/solr/response/TestRawTransformer.java
+++ b/solr/core/src/test/org/apache/solr/response/TestRawTransformer.java
@@ -26,6 +26,7 @@
 import java.util.regex.Pattern;
 import org.apache.solr.SolrTestCaseJ4;
 import org.apache.solr.client.solrj.SolrClient;
+import org.apache.solr.client.solrj.embedded.EmbeddedSolrServer;
 import org.apache.solr.client.solrj.impl.CloudSolrClient;
 import org.apache.solr.client.solrj.impl.NoOpResponseParser;
 import org.apache.solr.client.solrj.request.CollectionAdminRequest;
@@ -60,7 +61,11 @@ public static void beforeClass() throws Exception {
     if (random().nextBoolean()) {
       initStandalone();
       JSR.start();
-      CLIENT = JSR.newClient();
+      if (random().nextBoolean()) {
+        CLIENT = JSR.newClient();
+      } else {
+        CLIENT = new EmbeddedSolrServer(JSR.getCoreContainer(), null);
+      }
     } else {
       initCloud();
       CLIENT = JSR.newClient();
diff --git a/solr/solrj/src/java/org/apache/solr/client/solrj/SolrRequest.java b/solr/solrj/src/java/org/apache/solr/client/solrj/SolrRequest.java
index a615d478b98..58a1c239683 100644
--- a/solr/solrj/src/java/org/apache/solr/client/solrj/SolrRequest.java
+++ b/solr/solrj/src/java/org/apache/solr/client/solrj/SolrRequest.java
@@ -250,7 +250,8 @@ public final T process(SolrClient client, String collection)
       throws SolrServerException, IOException {
     long startNanos = System.nanoTime();
     T res = createResponse(client);
-    res.setResponse(client.request(this, collection));
+    var namedList = client.request(this, collection);
+    res.setResponse(namedList);
     long endNanos = System.nanoTime();
     res.setElapsedTime(TimeUnit.NANOSECONDS.toMillis(endNanos - startNanos));
     return res;
diff --git a/solr/solrj/src/java/org/apache/solr/client/solrj/impl/NoOpResponseParser.java b/solr/solrj/src/java/org/apache/solr/client/solrj/impl/NoOpResponseParser.java
index 919b6b873f1..92101918db5 100644
--- a/solr/solrj/src/java/org/apache/solr/client/solrj/impl/NoOpResponseParser.java
+++ b/solr/solrj/src/java/org/apache/solr/client/solrj/impl/NoOpResponseParser.java
@@ -26,8 +26,7 @@
 import org.apache.solr.common.util.NamedList;
 
 /**
- * Simply puts the entire response into an entry in a NamedList. This parser isn't parse response
- * into a QueryResponse.
+ * A special parser that puts the entire response into a string "response" field in the NamedList.
  */
 public class NoOpResponseParser extends ResponseParser {
 

From e58cc252eb475c05b549f9960eea4d1deb92dffe Mon Sep 17 00:00:00 2001
From: Jason Gerlowski <gerlowskija@apache.org>
Date: Tue, 5 Nov 2024 05:12:27 -0800
Subject: [PATCH 24/32] SOLR-17256: Remove usages of deprecated
 SolrRequest.setBasePath (#2811)

A previous commit deprecated SolrRequest.setBasePath but left it still
used throughout Solr's codebase.  This commit switches (almost) all
usage over to the safer `Http2SolrClient.requestWithBaseUrl`.

This will allow setBasePath to be removed more easily in the future.

---------

Co-authored-by: David Smiley <dsmiley@apache.org>
---
 .../apache/solr/bench/MiniClusterState.java   | 36 +++++++--------
 .../solr/bench/index/CloudIndexing.java       |  6 +--
 .../apache/solr/bench/search/FilterCache.java | 17 +++----
 .../solr/bench/search/JsonFaceting.java       | 10 +++--
 .../solr/bench/search/NumericSearch.java      |  8 ++--
 .../bench/search/QueryResponseWriters.java    |  1 -
 .../solr/bench/search/SimpleSearch.java       |  3 --
 .../solr/bench/MiniClusterBenchStateTest.java |  3 --
 .../org/apache/solr/cloud/ZkController.java   |  8 ++--
 .../org/apache/solr/handler/IndexFetcher.java |  3 +-
 .../solr/update/PeerSyncWithLeader.java       |  7 ++-
 .../solr/update/SolrCmdDistributor.java       |  5 ++-
 .../processor/DistributedUpdateProcessor.java |  7 ++-
 .../solr/handler/ReplicationTestHelper.java   |  8 ++--
 .../TestUserManagedReplicationWithAuth.java   | 14 +++---
 .../impl/SolrClientNodeStateProvider.java     |  6 +--
 .../common/cloud/NodesSysPropsCacher.java     | 12 ++---
 .../impl/ConcurrentUpdateHttp2SolrClient.java | 19 ++++----
 .../client/solrj/impl/LBHttp2SolrClient.java  | 29 +++++++-----
 .../solr/client/solrj/impl/LBSolrClient.java  | 44 +++++++++++++++----
 .../solrj/request/HealthCheckRequest.java     |  5 ++-
 .../solrj/impl/Http2SolrClientTest.java       | 16 ++++++-
 .../solrj/impl/HttpJdkSolrClientTest.java     |  9 ----
 .../solrj/impl/HttpSolrClientTestBase.java    | 28 ------------
 .../solrj/impl/LBHttp2SolrClientTest.java     | 28 ++++++++++--
 25 files changed, 177 insertions(+), 155 deletions(-)

diff --git a/solr/benchmark/src/java/org/apache/solr/bench/MiniClusterState.java b/solr/benchmark/src/java/org/apache/solr/bench/MiniClusterState.java
index 88bfc05af1f..34a970b3392 100755
--- a/solr/benchmark/src/java/org/apache/solr/bench/MiniClusterState.java
+++ b/solr/benchmark/src/java/org/apache/solr/bench/MiniClusterState.java
@@ -37,7 +37,6 @@
 import java.util.concurrent.Executors;
 import java.util.concurrent.ScheduledExecutorService;
 import java.util.concurrent.TimeUnit;
-import org.apache.solr.client.solrj.SolrClient;
 import org.apache.solr.client.solrj.SolrQuery;
 import org.apache.solr.client.solrj.SolrServerException;
 import org.apache.solr.client.solrj.impl.Http2SolrClient;
@@ -84,7 +83,7 @@ public static class MiniClusterBenchState {
     MiniSolrCloudCluster cluster;
 
     /** The Client. */
-    public SolrClient client;
+    public Http2SolrClient client;
 
     /** The Run cnt. */
     int runCnt = 0;
@@ -288,7 +287,7 @@ public void startMiniCluster(int nodeCount) {
       }
       zkHost = cluster.getZkServer().getZkAddress();
 
-      client = new Http2SolrClient.Builder().useHttp1_1(useHttp1).build();
+      client = new Http2SolrClient.Builder(nodes.get(0)).useHttp1_1(useHttp1).build();
 
       log("done starting mini cluster");
       log("");
@@ -318,9 +317,8 @@ public void createCollection(String collection, int numShards, int numReplicas)
 
           CollectionAdminRequest.Create request =
               CollectionAdminRequest.createCollection(collection, "conf", numShards, numReplicas);
-          request.setBasePath(nodes.get(random.nextInt(cluster.getJettySolrRunners().size())));
-
-          client.request(request);
+          client.requestWithBaseUrl(
+              nodes.get(random.nextInt(cluster.getJettySolrRunners().size())), null, request);
 
           cluster.waitForActiveCollection(
               collection, 15, TimeUnit.SECONDS, numShards, numShards * numReplicas);
@@ -368,18 +366,19 @@ public void index(String collection, Docs docs, int docCount, boolean parallel)
 
         log("committing data ...");
         UpdateRequest commitRequest = new UpdateRequest();
-        commitRequest.setBasePath(nodes.get(random.nextInt(cluster.getJettySolrRunners().size())));
+        final var url = nodes.get(random.nextInt(cluster.getJettySolrRunners().size()));
         commitRequest.setAction(UpdateRequest.ACTION.COMMIT, false, true);
-        commitRequest.process(client, collection);
+        client.requestWithBaseUrl(url, collection, commitRequest);
         log("done committing data");
       } else {
         cluster.waitForActiveCollection(collection, 15, TimeUnit.SECONDS);
       }
 
       QueryRequest queryRequest = new QueryRequest(new SolrQuery("q", "*:*", "rows", "1"));
-      queryRequest.setBasePath(nodes.get(random.nextInt(cluster.getJettySolrRunners().size())));
+      final var url = nodes.get(random.nextInt(cluster.getJettySolrRunners().size()));
+      NamedList<Object> result =
+          client.requestWithBaseUrl(url, collection, queryRequest).getResponse();
 
-      NamedList<Object> result = client.request(queryRequest, collection);
       log("sanity check of single row query result: " + result);
       log("");
 
@@ -417,15 +416,15 @@ private void indexParallel(String collection, Docs docs, int docCount)
               @Override
               public void run() {
                 UpdateRequest updateRequest = new UpdateRequest();
-                updateRequest.setBasePath(
-                    nodes.get(threadRandom.nextInt(cluster.getJettySolrRunners().size())));
+                final var url =
+                    nodes.get(threadRandom.nextInt(cluster.getJettySolrRunners().size()));
                 SolrInputDocument doc = docs.inputDocument();
                 // log("add doc " + doc);
                 updateRequest.add(doc);
                 meter.mark();
 
                 try {
-                  client.request(updateRequest, collection);
+                  client.requestWithBaseUrl(url, collection, updateRequest);
                 } catch (Exception e) {
                   throw new RuntimeException(e);
                 }
@@ -452,9 +451,8 @@ private void indexBatch(String collection, Docs docs, int docCount, int batchSiz
         batch.add(docs.inputDocument());
         if (i % batchSize == 0) {
           UpdateRequest updateRequest = new UpdateRequest();
-          updateRequest.setBasePath(nodes.get(0));
           updateRequest.add(batch);
-          client.request(updateRequest, collection);
+          client.requestWithBaseUrl(nodes.get(0), collection, updateRequest);
           meter.mark(batch.size());
           batch.clear();
           log(meter.getCount() + " docs at " + (long) meter.getMeanRate() + " doc/s");
@@ -462,9 +460,8 @@ private void indexBatch(String collection, Docs docs, int docCount, int batchSiz
       }
       if (!batch.isEmpty()) {
         UpdateRequest updateRequest = new UpdateRequest();
-        updateRequest.setBasePath(nodes.get(0));
         updateRequest.add(batch);
-        client.request(updateRequest, collection);
+        client.requestWithBaseUrl(nodes.get(0), collection, updateRequest);
         meter.mark(batch.size());
         batch = null;
       }
@@ -500,10 +497,9 @@ public void forceMerge(String collection, int maxMergeSegments) throws Exception
         }
 
         UpdateRequest optimizeRequest = new UpdateRequest();
-        optimizeRequest.setBasePath(
-            nodes.get(random.nextInt(cluster.getJettySolrRunners().size())));
+        final var url = nodes.get(random.nextInt(cluster.getJettySolrRunners().size()));
         optimizeRequest.setAction(UpdateRequest.ACTION.OPTIMIZE, false, true, maxMergeSegments);
-        optimizeRequest.process(client, collection);
+        client.requestWithBaseUrl(url, collection, optimizeRequest);
       }
     }
 
diff --git a/solr/benchmark/src/java/org/apache/solr/bench/index/CloudIndexing.java b/solr/benchmark/src/java/org/apache/solr/bench/index/CloudIndexing.java
index 820f2130a60..0ee950e7231 100755
--- a/solr/benchmark/src/java/org/apache/solr/bench/index/CloudIndexing.java
+++ b/solr/benchmark/src/java/org/apache/solr/bench/index/CloudIndexing.java
@@ -127,9 +127,9 @@ public void doSetup(MiniClusterState.MiniClusterBenchState miniClusterState) thr
   public Object indexDoc(MiniClusterState.MiniClusterBenchState miniClusterState, BenchState state)
       throws Exception {
     UpdateRequest updateRequest = new UpdateRequest();
-    updateRequest.setBasePath(
-        miniClusterState.nodes.get(miniClusterState.getRandom().nextInt(state.nodeCount)));
     updateRequest.add(state.getNextDoc());
-    return miniClusterState.client.request(updateRequest, BenchState.COLLECTION);
+    final var url =
+        miniClusterState.nodes.get(miniClusterState.getRandom().nextInt(state.nodeCount));
+    return miniClusterState.client.requestWithBaseUrl(url, BenchState.COLLECTION, updateRequest);
   }
 }
diff --git a/solr/benchmark/src/java/org/apache/solr/bench/search/FilterCache.java b/solr/benchmark/src/java/org/apache/solr/bench/search/FilterCache.java
index e518ff90865..6b7f6bf4a4f 100644
--- a/solr/benchmark/src/java/org/apache/solr/bench/search/FilterCache.java
+++ b/solr/benchmark/src/java/org/apache/solr/bench/search/FilterCache.java
@@ -68,6 +68,7 @@ public static class BenchState {
 
     QueryRequest q1 = new QueryRequest(new SolrQuery("q", "*:*", "fq", "Ea_b:true"));
     QueryRequest q2 = new QueryRequest(new SolrQuery("q", "*:*", "fq", "FB_b:true"));
+    String baseUrl;
 
     @Setup(Level.Trial)
     public void setupTrial(MiniClusterState.MiniClusterBenchState miniClusterState)
@@ -100,9 +101,7 @@ public Boolean generate(SolrRandomnessSource in) {
       docs.field("FB_b", booleans);
 
       miniClusterState.index(COLLECTION, docs, 30 * 1000);
-      String base = miniClusterState.nodes.get(0);
-      q1.setBasePath(base);
-      q2.setBasePath(base);
+      baseUrl = miniClusterState.nodes.get(0);
     }
 
     @Setup(Level.Iteration)
@@ -110,8 +109,7 @@ public void setupIteration(MiniClusterState.MiniClusterBenchState miniClusterSta
         throws SolrServerException, IOException {
       // Reload the collection/core to drop existing caches
       CollectionAdminRequest.Reload reload = CollectionAdminRequest.reloadCollection(COLLECTION);
-      reload.setBasePath(miniClusterState.nodes.get(0));
-      miniClusterState.client.request(reload);
+      miniClusterState.client.requestWithBaseUrl(miniClusterState.nodes.get(0), null, reload);
     }
 
     @TearDown(Level.Iteration)
@@ -139,14 +137,17 @@ public void dumpMetrics(MiniClusterState.MiniClusterBenchState miniClusterState)
   public Object filterCacheMultipleQueries(
       BenchState benchState, MiniClusterState.MiniClusterBenchState miniClusterState)
       throws SolrServerException, IOException {
-    return miniClusterState.client.request(
-        miniClusterState.getRandom().nextBoolean() ? benchState.q1 : benchState.q2, COLLECTION);
+    return miniClusterState.client.requestWithBaseUrl(
+        benchState.baseUrl,
+        COLLECTION,
+        miniClusterState.getRandom().nextBoolean() ? benchState.q1 : benchState.q2);
   }
 
   @Benchmark
   public Object filterCacheSingleQuery(
       BenchState benchState, MiniClusterState.MiniClusterBenchState miniClusterState)
       throws SolrServerException, IOException {
-    return miniClusterState.client.request(benchState.q1, COLLECTION);
+    return miniClusterState.client.requestWithBaseUrl(
+        benchState.baseUrl, COLLECTION, benchState.q1);
   }
 }
diff --git a/solr/benchmark/src/java/org/apache/solr/bench/search/JsonFaceting.java b/solr/benchmark/src/java/org/apache/solr/bench/search/JsonFaceting.java
index 498faed5814..1595de56d3b 100755
--- a/solr/benchmark/src/java/org/apache/solr/bench/search/JsonFaceting.java
+++ b/solr/benchmark/src/java/org/apache/solr/bench/search/JsonFaceting.java
@@ -180,11 +180,13 @@ public Object jsonFacet(
       BenchState state,
       BenchState.ThreadState threadState)
       throws Exception {
+    final var url = miniClusterState.nodes.get(threadState.random.nextInt(state.nodeCount));
     QueryRequest queryRequest = new QueryRequest(state.params);
-    queryRequest.setBasePath(
-        miniClusterState.nodes.get(threadState.random.nextInt(state.nodeCount)));
-
-    NamedList<Object> result = miniClusterState.client.request(queryRequest, state.collection);
+    NamedList<Object> result =
+        miniClusterState
+            .client
+            .requestWithBaseUrl(url, state.collection, queryRequest)
+            .getResponse();
 
     // MiniClusterState.log("result: " + result);
 
diff --git a/solr/benchmark/src/java/org/apache/solr/bench/search/NumericSearch.java b/solr/benchmark/src/java/org/apache/solr/bench/search/NumericSearch.java
index 3c7a72385f9..4d662205230 100644
--- a/solr/benchmark/src/java/org/apache/solr/bench/search/NumericSearch.java
+++ b/solr/benchmark/src/java/org/apache/solr/bench/search/NumericSearch.java
@@ -101,8 +101,8 @@ public void setupTrial(MiniClusterState.MiniClusterBenchState miniClusterState)
       q.setParam("facet.field", "numbers_i_dv", "term_low_s", "term_high_s");
       q.setParam("facet.limit", String.valueOf(maxCardinality));
       QueryRequest req = new QueryRequest(q);
-      req.setBasePath(basePath);
-      QueryResponse response = req.process(miniClusterState.client, COLLECTION);
+      QueryResponse response =
+          miniClusterState.client.requestWithBaseUrl(basePath, COLLECTION, req);
       Set<String> numbers =
           response.getFacetField("numbers_i_dv").getValues().stream()
               .map(FacetField.Count::getName)
@@ -144,8 +144,7 @@ public void setupIteration(MiniClusterState.MiniClusterBenchState miniClusterSta
         throws SolrServerException, IOException {
       // Reload the collection/core to drop existing caches
       CollectionAdminRequest.Reload reload = CollectionAdminRequest.reloadCollection(COLLECTION);
-      reload.setBasePath(miniClusterState.nodes.get(0));
-      miniClusterState.client.request(reload);
+      miniClusterState.client.requestWithBaseUrl(miniClusterState.nodes.get(0), null, reload);
     }
 
     public QueryRequest intSetQuery(boolean dvs) {
@@ -172,7 +171,6 @@ QueryRequest setQuery(String field) {
                   termQueryField + ":" + lowCardTerms.next(),
                   "fq",
                   "{!terms cache=false f='" + field + "'}" + queries.next()));
-      q.setBasePath(basePath);
       return q;
     }
   }
diff --git a/solr/benchmark/src/java/org/apache/solr/bench/search/QueryResponseWriters.java b/solr/benchmark/src/java/org/apache/solr/bench/search/QueryResponseWriters.java
index 3c3217cc085..111a1d6ac50 100644
--- a/solr/benchmark/src/java/org/apache/solr/bench/search/QueryResponseWriters.java
+++ b/solr/benchmark/src/java/org/apache/solr/bench/search/QueryResponseWriters.java
@@ -88,7 +88,6 @@ public void setup(MiniClusterBenchState miniClusterState) throws Exception {
       q = new QueryRequest(params);
       q.setResponseParser(new NoOpResponseParser(wt));
       String base = miniClusterState.nodes.get(0);
-      q.setBasePath(base);
     }
   }
 
diff --git a/solr/benchmark/src/java/org/apache/solr/bench/search/SimpleSearch.java b/solr/benchmark/src/java/org/apache/solr/bench/search/SimpleSearch.java
index 165d68ecb3b..46982b33b89 100644
--- a/solr/benchmark/src/java/org/apache/solr/bench/search/SimpleSearch.java
+++ b/solr/benchmark/src/java/org/apache/solr/bench/search/SimpleSearch.java
@@ -65,8 +65,6 @@ public void setupTrial(MiniClusterState.MiniClusterBenchState miniClusterState)
       miniClusterState.setUseHttp1(useHttp1);
       miniClusterState.startMiniCluster(1);
       miniClusterState.createCollection(COLLECTION, 1, 1);
-      String base = miniClusterState.nodes.get(0);
-      q.setBasePath(base);
     }
 
     @Setup(Level.Iteration)
@@ -74,7 +72,6 @@ public void setupIteration(MiniClusterState.MiniClusterBenchState miniClusterSta
         throws SolrServerException, IOException {
       // Reload the collection/core to drop existing caches
       CollectionAdminRequest.Reload reload = CollectionAdminRequest.reloadCollection(COLLECTION);
-      reload.setBasePath(miniClusterState.nodes.get(0));
       miniClusterState.client.request(reload);
 
       total = new AtomicLong();
diff --git a/solr/benchmark/src/test/org/apache/solr/bench/MiniClusterBenchStateTest.java b/solr/benchmark/src/test/org/apache/solr/bench/MiniClusterBenchStateTest.java
index e17ed29dcd8..af568e3bb98 100644
--- a/solr/benchmark/src/test/org/apache/solr/bench/MiniClusterBenchStateTest.java
+++ b/solr/benchmark/src/test/org/apache/solr/bench/MiniClusterBenchStateTest.java
@@ -118,10 +118,7 @@ public void testMiniClusterState() throws Exception {
     miniBenchState.forceMerge(collection, 15);
 
     ModifiableSolrParams params = MiniClusterState.params("q", "*:*");
-
     QueryRequest queryRequest = new QueryRequest(params);
-    queryRequest.setBasePath(miniBenchState.nodes.get(0));
-
     QueryResponse result = queryRequest.process(miniBenchState.client, collection);
 
     BaseBenchState.log("match all query result=" + result);
diff --git a/solr/core/src/java/org/apache/solr/cloud/ZkController.java b/solr/core/src/java/org/apache/solr/cloud/ZkController.java
index 59faced1b84..25c5cb1fa9e 100644
--- a/solr/core/src/java/org/apache/solr/cloud/ZkController.java
+++ b/solr/core/src/java/org/apache/solr/cloud/ZkController.java
@@ -396,10 +396,10 @@ public ZkController(
     }
     this.overseerCollectionQueue = overseer.getCollectionQueue(zkClient);
     this.overseerConfigSetQueue = overseer.getConfigSetQueue(zkClient);
-    this.sysPropsCacher =
-        new NodesSysPropsCacher(
-            ((HttpShardHandlerFactory) getCoreContainer().getShardHandlerFactory()).getClient(),
-            zkStateReader);
+    final var client =
+        (Http2SolrClient)
+            ((HttpShardHandlerFactory) getCoreContainer().getShardHandlerFactory()).getClient();
+    this.sysPropsCacher = new NodesSysPropsCacher(client, zkStateReader);
     assert ObjectReleaseTracker.track(this);
   }
 
diff --git a/solr/core/src/java/org/apache/solr/handler/IndexFetcher.java b/solr/core/src/java/org/apache/solr/handler/IndexFetcher.java
index 8cce5ed129f..66acda5c185 100644
--- a/solr/core/src/java/org/apache/solr/handler/IndexFetcher.java
+++ b/solr/core/src/java/org/apache/solr/handler/IndexFetcher.java
@@ -1980,9 +1980,8 @@ private FastInputStream getStream() throws IOException {
       try {
         QueryRequest req = new QueryRequest(params);
         req.setResponseParser(new InputStreamResponseParser(FILE_STREAM));
-        req.setBasePath(leaderBaseUrl);
         if (useExternalCompression) req.addHeader("Accept-Encoding", "gzip");
-        response = solrClient.request(req, leaderCoreName);
+        response = solrClient.requestWithBaseUrl(leaderBaseUrl, leaderCoreName, req).getResponse();
         final var responseStatus = (Integer) response.get("responseStatus");
         is = (InputStream) response.get("stream");
 
diff --git a/solr/core/src/java/org/apache/solr/update/PeerSyncWithLeader.java b/solr/core/src/java/org/apache/solr/update/PeerSyncWithLeader.java
index ad7c76d752a..f46e0c161e7 100644
--- a/solr/core/src/java/org/apache/solr/update/PeerSyncWithLeader.java
+++ b/solr/core/src/java/org/apache/solr/update/PeerSyncWithLeader.java
@@ -28,9 +28,9 @@
 import java.lang.invoke.MethodHandles;
 import java.util.List;
 import java.util.Set;
-import org.apache.solr.client.solrj.SolrClient;
 import org.apache.solr.client.solrj.SolrRequest;
 import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.client.solrj.impl.Http2SolrClient;
 import org.apache.solr.client.solrj.request.QueryRequest;
 import org.apache.solr.client.solrj.response.QueryResponse;
 import org.apache.solr.cloud.ZkController;
@@ -54,7 +54,7 @@ public class PeerSyncWithLeader implements SolrMetricProducer {
 
   private UpdateHandler uhandler;
   private UpdateLog ulog;
-  private final SolrClient clientToLeader;
+  private final Http2SolrClient clientToLeader;
   private final String coreName;
   private final String leaderBaseUrl;
 
@@ -334,8 +334,7 @@ private boolean handleUpdates(
   private NamedList<Object> request(ModifiableSolrParams params, String onFail) {
     try {
       QueryRequest request = new QueryRequest(params, SolrRequest.METHOD.POST);
-      request.setBasePath(leaderBaseUrl);
-      QueryResponse rsp = request.process(clientToLeader, coreName);
+      QueryResponse rsp = clientToLeader.requestWithBaseUrl(leaderBaseUrl, coreName, request);
       Exception exception = rsp.getException();
       if (exception != null) {
         throw new SolrException(SolrException.ErrorCode.SERVER_ERROR, onFail);
diff --git a/solr/core/src/java/org/apache/solr/update/SolrCmdDistributor.java b/solr/core/src/java/org/apache/solr/update/SolrCmdDistributor.java
index d31ddc9ca55..7f1998d6c82 100644
--- a/solr/core/src/java/org/apache/solr/update/SolrCmdDistributor.java
+++ b/solr/core/src/java/org/apache/solr/update/SolrCmdDistributor.java
@@ -315,8 +315,9 @@ private void submit(final Req req, boolean isCommit) throws IOException {
       blockAndDoRetries();
 
       try {
-        req.uReq.setBasePath(req.node.getUrl());
-        clients.getHttpClient().request(req.uReq);
+        clients
+            .getHttpClient()
+            .requestWithBaseUrl(req.node.getBaseUrl(), req.node.getCoreName(), req.uReq);
       } catch (Exception e) {
         log.error("Exception making request", e);
         SolrError error = new SolrError();
diff --git a/solr/core/src/java/org/apache/solr/update/processor/DistributedUpdateProcessor.java b/solr/core/src/java/org/apache/solr/update/processor/DistributedUpdateProcessor.java
index d8c88ac5786..8e3153046ad 100644
--- a/solr/core/src/java/org/apache/solr/update/processor/DistributedUpdateProcessor.java
+++ b/solr/core/src/java/org/apache/solr/update/processor/DistributedUpdateProcessor.java
@@ -692,8 +692,11 @@ private UpdateCommand fetchFullUpdateFromLeader(AddUpdateCommand inplaceAdd, lon
 
     NamedList<Object> rsp;
     try {
-      ur.setBasePath(leaderUrl);
-      rsp = updateShardHandler.getUpdateOnlyHttpClient().request(ur);
+      rsp =
+          updateShardHandler
+              .getUpdateOnlyHttpClient()
+              .requestWithBaseUrl(leaderUrl, null, ur)
+              .getResponse();
     } catch (SolrServerException e) {
       throw new SolrException(
           ErrorCode.SERVER_ERROR,
diff --git a/solr/core/src/test/org/apache/solr/handler/ReplicationTestHelper.java b/solr/core/src/test/org/apache/solr/handler/ReplicationTestHelper.java
index edd68727f87..2e69ba4915a 100644
--- a/solr/core/src/test/org/apache/solr/handler/ReplicationTestHelper.java
+++ b/solr/core/src/test/org/apache/solr/handler/ReplicationTestHelper.java
@@ -33,7 +33,7 @@
 import org.apache.solr.SolrTestCaseJ4;
 import org.apache.solr.client.solrj.SolrClient;
 import org.apache.solr.client.solrj.SolrServerException;
-import org.apache.solr.client.solrj.impl.HttpSolrClient;
+import org.apache.solr.client.solrj.impl.Http2SolrClient;
 import org.apache.solr.client.solrj.request.QueryRequest;
 import org.apache.solr.client.solrj.response.QueryResponse;
 import org.apache.solr.common.SolrDocumentList;
@@ -76,11 +76,11 @@ public static SolrClient createNewSolrClient(String baseUrl) {
    * @param baseUrl the root URL for a Solr node
    * @param collectionOrCore an optional default collection/core for the created client
    */
-  public static SolrClient createNewSolrClient(String baseUrl, String collectionOrCore) {
-    return new HttpSolrClient.Builder(baseUrl)
+  public static Http2SolrClient createNewSolrClient(String baseUrl, String collectionOrCore) {
+    return new Http2SolrClient.Builder(baseUrl)
         .withDefaultCollection(collectionOrCore)
         .withConnectionTimeout(15000, TimeUnit.MILLISECONDS)
-        .withSocketTimeout(90000, TimeUnit.MILLISECONDS)
+        .withIdleTimeout(90000, TimeUnit.MILLISECONDS)
         .build();
   }
 
diff --git a/solr/core/src/test/org/apache/solr/handler/TestUserManagedReplicationWithAuth.java b/solr/core/src/test/org/apache/solr/handler/TestUserManagedReplicationWithAuth.java
index b230aad2021..98a5684791d 100644
--- a/solr/core/src/test/org/apache/solr/handler/TestUserManagedReplicationWithAuth.java
+++ b/solr/core/src/test/org/apache/solr/handler/TestUserManagedReplicationWithAuth.java
@@ -33,6 +33,7 @@
 import org.apache.solr.client.solrj.SolrRequest;
 import org.apache.solr.client.solrj.SolrResponse;
 import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.client.solrj.impl.Http2SolrClient;
 import org.apache.solr.client.solrj.request.HealthCheckRequest;
 import org.apache.solr.client.solrj.request.QueryRequest;
 import org.apache.solr.client.solrj.request.UpdateRequest;
@@ -48,7 +49,7 @@
 @SuppressSSL
 public class TestUserManagedReplicationWithAuth extends SolrTestCaseJ4 {
   JettySolrRunner leaderJetty, followerJetty, followerJettyWithAuth;
-  SolrClient leaderClient, followerClient, followerClientWithAuth;
+  Http2SolrClient leaderClient, followerClient, followerClientWithAuth;
   ReplicationTestHelper.SolrInstance leader = null, follower = null, followerWithAuth = null;
 
   private static String user = "solr";
@@ -227,7 +228,7 @@ private QueryResponse queryWithBasicAuth(SolrClient client, SolrQuery q)
     return withBasicAuth(new QueryRequest(q)).process(client);
   }
 
-  private void disablePoll(JettySolrRunner Jetty, SolrClient solrClient)
+  private void disablePoll(JettySolrRunner Jetty, Http2SolrClient solrClient)
       throws SolrServerException, IOException {
     ModifiableSolrParams disablePollParams = new ModifiableSolrParams();
     disablePollParams.set(COMMAND, CMD_DISABLE_POLL);
@@ -235,19 +236,18 @@ private void disablePoll(JettySolrRunner Jetty, SolrClient solrClient)
     disablePollParams.set(CommonParams.QT, ReplicationHandler.PATH);
     QueryRequest req = new QueryRequest(disablePollParams);
     withBasicAuth(req);
-    req.setBasePath(buildUrl(Jetty.getLocalPort()));
 
-    solrClient.request(req, DEFAULT_TEST_CORENAME);
+    final var baseUrl = buildUrl(Jetty.getLocalPort());
+    solrClient.requestWithBaseUrl(baseUrl, DEFAULT_TEST_CORENAME, req);
   }
 
   private void pullIndexFromTo(
       JettySolrRunner srcSolr, JettySolrRunner destSolr, boolean authEnabled)
       throws SolrServerException, IOException {
     String srcUrl = buildUrl(srcSolr.getLocalPort()) + "/" + DEFAULT_TEST_CORENAME;
-    String destUrl = buildUrl(destSolr.getLocalPort()) + "/" + DEFAULT_TEST_CORENAME;
     QueryRequest req = getQueryRequestForFetchIndex(authEnabled, srcUrl);
-    req.setBasePath(buildUrl(destSolr.getLocalPort()));
-    followerClient.request(req, DEFAULT_TEST_CORENAME);
+    final var baseUrl = buildUrl(destSolr.getLocalPort());
+    followerClient.requestWithBaseUrl(baseUrl, DEFAULT_TEST_CORENAME, req);
   }
 
   private QueryRequest getQueryRequestForFetchIndex(boolean authEnabled, String srcUrl) {
diff --git a/solr/solrj-zookeeper/src/java/org/apache/solr/client/solrj/impl/SolrClientNodeStateProvider.java b/solr/solrj-zookeeper/src/java/org/apache/solr/client/solrj/impl/SolrClientNodeStateProvider.java
index 984e1b7a18f..a9789c8a141 100644
--- a/solr/solrj-zookeeper/src/java/org/apache/solr/client/solrj/impl/SolrClientNodeStateProvider.java
+++ b/solr/solrj-zookeeper/src/java/org/apache/solr/client/solrj/impl/SolrClientNodeStateProvider.java
@@ -44,7 +44,6 @@
 import org.apache.solr.common.params.CommonParams;
 import org.apache.solr.common.params.ModifiableSolrParams;
 import org.apache.solr.common.params.SolrParams;
-import org.apache.solr.common.util.NamedList;
 import org.apache.solr.common.util.Pair;
 import org.apache.solr.common.util.Utils;
 import org.slf4j.Logger;
@@ -293,10 +292,7 @@ public SimpleSolrResponse invoke(String solrNode, String path, SolrParams params
       request.setResponseParser(new BinaryResponseParser());
 
       try {
-        NamedList<Object> rsp =
-            cloudSolrClient.getHttpClient().requestWithBaseUrl(url, request::process).getResponse();
-        request.response.setResponse(rsp);
-        return request.response;
+        return cloudSolrClient.getHttpClient().requestWithBaseUrl(url, request::process);
       } catch (SolrServerException | IOException e) {
         throw new SolrException(ErrorCode.SERVER_ERROR, "Fetching replica metrics failed", e);
       }
diff --git a/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/NodesSysPropsCacher.java b/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/NodesSysPropsCacher.java
index df9a7c5dbde..37699901323 100644
--- a/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/NodesSysPropsCacher.java
+++ b/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/NodesSysPropsCacher.java
@@ -21,8 +21,8 @@
 import java.util.LinkedHashMap;
 import java.util.Map;
 import java.util.concurrent.ConcurrentHashMap;
-import org.apache.solr.client.solrj.SolrClient;
 import org.apache.solr.client.solrj.SolrRequest;
+import org.apache.solr.client.solrj.impl.Http2SolrClient;
 import org.apache.solr.client.solrj.request.GenericSolrRequest;
 import org.apache.solr.common.MapWriter;
 import org.apache.solr.common.NavigableObject;
@@ -35,9 +35,9 @@ public class NodesSysPropsCacher implements NodesSysProps, AutoCloseable {
   private volatile boolean isClosed = false;
   private final Map<String, Map<String, Object>> nodeVsTagsCache = new ConcurrentHashMap<>();
   private ZkStateReader zkStateReader;
-  private final SolrClient solrClient;
+  private final Http2SolrClient solrClient;
 
-  public NodesSysPropsCacher(SolrClient solrClient, ZkStateReader zkStateReader) {
+  public NodesSysPropsCacher(Http2SolrClient solrClient, ZkStateReader zkStateReader) {
     this.zkStateReader = zkStateReader;
     this.solrClient = solrClient;
     zkStateReader.registerLiveNodesListener(
@@ -86,10 +86,12 @@ private Map<String, Object> fetchProps(String nodeName, Collection<String> tags)
     }
 
     GenericSolrRequest req = new GenericSolrRequest(SolrRequest.METHOD.GET, "/admin/metrics", msp);
-    req.setBasePath(zkStateReader.getBaseUrlForNodeName(nodeName));
     try {
       LinkedHashMap<String, Object> result = new LinkedHashMap<>();
-      NavigableObject response = solrClient.request(req);
+      NavigableObject response =
+          solrClient
+              .requestWithBaseUrl(zkStateReader.getBaseUrlForNodeName(nodeName), null, req)
+              .getResponse();
       NavigableObject metrics = (NavigableObject) response._get("metrics", MapWriter.EMPTY);
       keys.forEach((tag, key) -> result.put(tag, metrics._get(key, null)));
       return result;
diff --git a/solr/solrj/src/java/org/apache/solr/client/solrj/impl/ConcurrentUpdateHttp2SolrClient.java b/solr/solrj/src/java/org/apache/solr/client/solrj/impl/ConcurrentUpdateHttp2SolrClient.java
index 28e595e21ec..7b44264f995 100644
--- a/solr/solrj/src/java/org/apache/solr/client/solrj/impl/ConcurrentUpdateHttp2SolrClient.java
+++ b/solr/solrj/src/java/org/apache/solr/client/solrj/impl/ConcurrentUpdateHttp2SolrClient.java
@@ -358,14 +358,14 @@ private void addRunner() {
   @Override
   public NamedList<Object> request(final SolrRequest<?> request, String collection)
       throws SolrServerException, IOException {
-    if (ClientUtils.shouldApplyDefaultCollection(collection, request))
-      collection = defaultCollection;
+    final String effectiveCollection =
+        ClientUtils.shouldApplyDefaultCollection(collection, request)
+            ? defaultCollection
+            : collection;
     if (!(request instanceof UpdateRequest)) {
-      request.setBasePath(basePath);
-      return client.request(request, collection);
+      return client.requestWithBaseUrl(basePath, (c) -> c.request(request, effectiveCollection));
     }
     UpdateRequest req = (UpdateRequest) request;
-    req.setBasePath(basePath);
     // this happens for commit...
     if (streamDeletes) {
       if ((req.getDocuments() == null || req.getDocuments().isEmpty())
@@ -373,13 +373,14 @@ public NamedList<Object> request(final SolrRequest<?> request, String collection
           && (req.getDeleteByIdMap() == null || req.getDeleteByIdMap().isEmpty())) {
         if (req.getDeleteQuery() == null) {
           blockUntilFinished();
-          return client.request(request, collection);
+          return client.requestWithBaseUrl(
+              basePath, (c) -> c.request(request, effectiveCollection));
         }
       }
     } else {
       if ((req.getDocuments() == null || req.getDocuments().isEmpty())) {
         blockUntilFinished();
-        return client.request(request, collection);
+        return client.requestWithBaseUrl(basePath, (c) -> c.request(request, effectiveCollection));
       }
     }
 
@@ -389,7 +390,7 @@ public NamedList<Object> request(final SolrRequest<?> request, String collection
       if (params.getBool(UpdateParams.WAIT_SEARCHER, false)) {
         log.info("blocking for commit/optimize");
         blockUntilFinished(); // empty the queue
-        return client.request(request, collection);
+        return client.requestWithBaseUrl(basePath, (c) -> c.request(request, effectiveCollection));
       }
     }
 
@@ -399,7 +400,7 @@ public NamedList<Object> request(final SolrRequest<?> request, String collection
         tmpLock.await();
       }
 
-      Update update = new Update(req, collection);
+      Update update = new Update(req, effectiveCollection);
       boolean success = queue.offer(update);
 
       long lastStallTime = -1;
diff --git a/solr/solrj/src/java/org/apache/solr/client/solrj/impl/LBHttp2SolrClient.java b/solr/solrj/src/java/org/apache/solr/client/solrj/impl/LBHttp2SolrClient.java
index e6633aff4d5..cb06c9d0c27 100644
--- a/solr/solrj/src/java/org/apache/solr/client/solrj/impl/LBHttp2SolrClient.java
+++ b/solr/solrj/src/java/org/apache/solr/client/solrj/impl/LBHttp2SolrClient.java
@@ -209,18 +209,23 @@ private CompletableFuture<NamedList<Object>> doAsyncRequest(
       RetryListener listener) {
     String baseUrl = endpoint.toString();
     rsp.server = baseUrl;
-    req.getRequest().setBasePath(baseUrl);
-    CompletableFuture<NamedList<Object>> future =
-        ((Http2SolrClient) getClient(endpoint)).requestAsync(req.getRequest());
-    future.whenComplete(
-        (result, throwable) -> {
-          if (!future.isCompletedExceptionally()) {
-            onSuccessfulRequest(result, endpoint, rsp, isZombie, listener);
-          } else if (!future.isCancelled()) {
-            onFailedRequest(throwable, endpoint, isNonRetryable, isZombie, listener);
-          }
-        });
-    return future;
+    final var client = (Http2SolrClient) getClient(endpoint);
+    try {
+      CompletableFuture<NamedList<Object>> future =
+          client.requestWithBaseUrl(baseUrl, (c) -> c.requestAsync(req.getRequest()));
+      future.whenComplete(
+          (result, throwable) -> {
+            if (!future.isCompletedExceptionally()) {
+              onSuccessfulRequest(result, endpoint, rsp, isZombie, listener);
+            } else if (!future.isCancelled()) {
+              onFailedRequest(throwable, endpoint, isNonRetryable, isZombie, listener);
+            }
+          });
+      return future;
+    } catch (SolrServerException | IOException e) {
+      // Unreachable, since 'requestWithBaseUrl' above is running the request asynchronously
+      throw new RuntimeException(e);
+    }
   }
 
   private void onSuccessfulRequest(
diff --git a/solr/solrj/src/java/org/apache/solr/client/solrj/impl/LBSolrClient.java b/solr/solrj/src/java/org/apache/solr/client/solrj/impl/LBSolrClient.java
index 795f2a003cb..ea5ecf396f7 100644
--- a/solr/solrj/src/java/org/apache/solr/client/solrj/impl/LBSolrClient.java
+++ b/solr/solrj/src/java/org/apache/solr/client/solrj/impl/LBSolrClient.java
@@ -480,14 +480,36 @@ private static boolean isTimeExceeded(long timeAllowedNano, long timeOutTime) {
     return timeAllowedNano > 0 && System.nanoTime() > timeOutTime;
   }
 
+  private NamedList<Object> doRequest(Endpoint endpoint, SolrRequest<?> solrRequest)
+      throws SolrServerException, IOException {
+    final var solrClient = getClient(endpoint);
+    return doRequest(solrClient, endpoint.getBaseUrl(), endpoint.getCore(), solrRequest);
+  }
+
+  // TODO SOLR-17541 should remove the need for the special-casing below; remove as a part of that
+  // ticket.
+  private NamedList<Object> doRequest(
+      SolrClient solrClient, String baseUrl, String collection, SolrRequest<?> solrRequest)
+      throws SolrServerException, IOException {
+    // Some implementations of LBSolrClient.getClient(...) return a Http2SolrClient that may not be
+    // pointed at the desired URL (or any URL for that matter).  We special case that here to ensure
+    // the appropriate URL is provided.
+    if (solrClient instanceof Http2SolrClient) {
+      final var httpSolrClient = (Http2SolrClient) solrClient;
+      return httpSolrClient.requestWithBaseUrl(baseUrl, (c) -> c.request(solrRequest, collection));
+    }
+
+    // Assume provided client already uses 'baseUrl'
+    return solrClient.request(solrRequest, collection);
+  }
+
   protected Exception doRequest(
       Endpoint baseUrl, Req req, Rsp rsp, boolean isNonRetryable, boolean isZombie)
       throws SolrServerException, IOException {
     Exception ex = null;
     try {
       rsp.server = baseUrl.toString();
-      req.getRequest().setBasePath(baseUrl.getBaseUrl());
-      rsp.rsp = getClient(baseUrl).request(req.getRequest(), baseUrl.getCore());
+      rsp.rsp = doRequest(baseUrl, req.getRequest());
       if (isZombie) {
         zombieServers.remove(baseUrl.toString());
       }
@@ -585,11 +607,18 @@ private void checkAZombieServer(EndpointWrapper zombieServer) {
     final Endpoint zombieEndpoint = zombieServer.getEndpoint();
     try {
       QueryRequest queryRequest = new QueryRequest(solrQuery);
-      queryRequest.setBasePath(zombieEndpoint.getBaseUrl());
       // First the one on the endpoint, then the default collection
       final String effectiveCollection =
           Objects.requireNonNullElse(zombieEndpoint.getCore(), getDefaultCollection());
-      QueryResponse resp = queryRequest.process(getClient(zombieEndpoint), effectiveCollection);
+      final var responseRaw =
+          doRequest(
+              getClient(zombieEndpoint),
+              zombieEndpoint.getBaseUrl(),
+              effectiveCollection,
+              queryRequest);
+      QueryResponse resp = new QueryResponse();
+      resp.setResponse(responseRaw);
+
       if (resp.getStatus() == 0) {
         // server has come back up.
         // make sure to remove from zombies before adding to the alive list to avoid a race
@@ -687,11 +716,10 @@ public NamedList<Object> request(
       final var endpoint = wrapper.getEndpoint();
       try {
         ++numServersTried;
-        request.setBasePath(endpoint.getBaseUrl());
         // Choose the endpoint's core/collection over any specified by the user
         final var effectiveCollection =
             endpoint.getCore() == null ? collection : endpoint.getCore();
-        return getClient(endpoint).request(request, effectiveCollection);
+        return doRequest(getClient(endpoint), endpoint.getBaseUrl(), effectiveCollection, request);
       } catch (SolrException e) {
         // Server is alive but the request was malformed or invalid
         throw e;
@@ -721,10 +749,10 @@ public NamedList<Object> request(
           || (justFailed != null && justFailed.containsKey(endpoint.getUrl()))) continue;
       try {
         ++numServersTried;
-        request.setBasePath(endpoint.getBaseUrl());
         final String effectiveCollection =
             endpoint.getCore() == null ? collection : endpoint.getCore();
-        NamedList<Object> rsp = getClient(endpoint).request(request, effectiveCollection);
+        NamedList<Object> rsp =
+            doRequest(getClient(endpoint), endpoint.getBaseUrl(), effectiveCollection, request);
         // remove from zombie list *before* adding to the alive list to avoid a race that could lose
         // a server
         zombieServers.remove(endpoint.getUrl());
diff --git a/solr/solrj/src/java/org/apache/solr/client/solrj/request/HealthCheckRequest.java b/solr/solrj/src/java/org/apache/solr/client/solrj/request/HealthCheckRequest.java
index 7937beadd02..77a7165bdf5 100644
--- a/solr/solrj/src/java/org/apache/solr/client/solrj/request/HealthCheckRequest.java
+++ b/solr/solrj/src/java/org/apache/solr/client/solrj/request/HealthCheckRequest.java
@@ -22,7 +22,8 @@
 import java.util.OptionalInt;
 import org.apache.solr.client.solrj.SolrClient;
 import org.apache.solr.client.solrj.SolrRequest;
-import org.apache.solr.client.solrj.impl.HttpSolrClient;
+import org.apache.solr.client.solrj.impl.BaseHttpSolrClient;
+import org.apache.solr.client.solrj.impl.HttpSolrClientBase;
 import org.apache.solr.client.solrj.response.HealthCheckResponse;
 import org.apache.solr.common.params.ModifiableSolrParams;
 import org.apache.solr.common.params.SolrParams;
@@ -58,7 +59,7 @@ public SolrParams getParams() {
   protected HealthCheckResponse createResponse(SolrClient client) {
     // TODO: Accept requests w/ CloudSolrClient while ensuring that the request doesn't get routed
     // to an unintended recipient.
-    assert client instanceof HttpSolrClient;
+    assert client instanceof BaseHttpSolrClient || client instanceof HttpSolrClientBase;
     return new HealthCheckResponse();
   }
 
diff --git a/solr/solrj/src/test/org/apache/solr/client/solrj/impl/Http2SolrClientTest.java b/solr/solrj/src/test/org/apache/solr/client/solrj/impl/Http2SolrClientTest.java
index 2474c772659..5927811906c 100644
--- a/solr/solrj/src/test/org/apache/solr/client/solrj/impl/Http2SolrClientTest.java
+++ b/solr/solrj/src/test/org/apache/solr/client/solrj/impl/Http2SolrClientTest.java
@@ -30,6 +30,8 @@
 import org.apache.solr.client.solrj.SolrServerException;
 import org.apache.solr.client.solrj.request.QueryRequest;
 import org.apache.solr.client.solrj.request.RequestWriter;
+import org.apache.solr.client.solrj.request.SolrPing;
+import org.apache.solr.common.SolrException;
 import org.apache.solr.common.params.CommonParams;
 import org.apache.solr.common.params.MapSolrParams;
 import org.apache.solr.common.params.ModifiableSolrParams;
@@ -118,10 +120,22 @@ public void testSolrExceptionCodeNotFromSolr() throws IOException, SolrServerExc
     }
   }
 
+  /** test that temporary baseURL overrides are reflected in error messages */
   @Test
   public void testSolrExceptionWithNullBaseurl() throws IOException, SolrServerException {
+    final int status = 527;
+    DebugServlet.setErrorCode(status);
+
     try (Http2SolrClient client = new Http2SolrClient.Builder(null).build()) {
-      super.testSolrExceptionWithNullBaseurl(client);
+      try {
+        // if client base url is null, request url will be used in exception message
+        client.requestWithBaseUrl(getBaseUrl() + DEBUG_SERVLET_PATH, DEFAULT_CORE, new SolrPing());
+
+        fail("Didn't get excepted exception from oversided request");
+      } catch (SolrException e) {
+        assertEquals("Unexpected exception status code", status, e.code());
+        assertTrue(e.getMessage().contains(getBaseUrl()));
+      }
     } finally {
       DebugServlet.clear();
     }
diff --git a/solr/solrj/src/test/org/apache/solr/client/solrj/impl/HttpJdkSolrClientTest.java b/solr/solrj/src/test/org/apache/solr/client/solrj/impl/HttpJdkSolrClientTest.java
index 698658a8528..c3fb5609476 100644
--- a/solr/solrj/src/test/org/apache/solr/client/solrj/impl/HttpJdkSolrClientTest.java
+++ b/solr/solrj/src/test/org/apache/solr/client/solrj/impl/HttpJdkSolrClientTest.java
@@ -305,15 +305,6 @@ public void testSolrExceptionCodeNotFromSolr() throws IOException, SolrServerExc
     }
   }
 
-  @Test
-  public void testSolrExceptionWithNullBaseurl() throws IOException, SolrServerException {
-    try (HttpJdkSolrClient client = builder(null).build()) {
-      super.testSolrExceptionWithNullBaseurl(client);
-    } finally {
-      DebugServlet.clear();
-    }
-  }
-
   @Test
   public void testUpdateDefault() throws Exception {
     String url = getBaseUrl() + DEBUG_SERVLET_PATH;
diff --git a/solr/solrj/src/test/org/apache/solr/client/solrj/impl/HttpSolrClientTestBase.java b/solr/solrj/src/test/org/apache/solr/client/solrj/impl/HttpSolrClientTestBase.java
index b2e51cbc63f..1c8525b74b6 100644
--- a/solr/solrj/src/test/org/apache/solr/client/solrj/impl/HttpSolrClientTestBase.java
+++ b/solr/solrj/src/test/org/apache/solr/client/solrj/impl/HttpSolrClientTestBase.java
@@ -42,7 +42,6 @@
 import org.apache.solr.client.solrj.SolrRequest;
 import org.apache.solr.client.solrj.SolrServerException;
 import org.apache.solr.client.solrj.request.QueryRequest;
-import org.apache.solr.client.solrj.request.SolrPing;
 import org.apache.solr.client.solrj.request.UpdateRequest;
 import org.apache.solr.client.solrj.response.QueryResponse;
 import org.apache.solr.common.SolrDocumentList;
@@ -251,33 +250,6 @@ public void testSolrExceptionCodeNotFromSolr(HttpSolrClientBase client)
     }
   }
 
-  /**
-   * test that SolrExceptions thrown by HttpSolrClient can correctly encapsulate http status codes
-   * even when not on the list of ErrorCodes solr may return.
-   */
-  public void testSolrExceptionWithNullBaseurl(HttpSolrClientBase client)
-      throws IOException, SolrServerException {
-    final int status = 527;
-    assertEquals(
-        status
-            + " didn't generate an UNKNOWN error code, someone modified the list of valid ErrorCode's w/o changing this test to work a different way",
-        SolrException.ErrorCode.UNKNOWN,
-        SolrException.ErrorCode.getErrorCode(status));
-
-    DebugServlet.setErrorCode(status);
-    try {
-      // if client base url is null, request url will be used in exception message
-      SolrPing ping = new SolrPing();
-      ping.setBasePath(getBaseUrl() + DEBUG_SERVLET_PATH);
-      client.request(ping, DEFAULT_CORE);
-
-      fail("Didn't get excepted exception from oversided request");
-    } catch (SolrException e) {
-      assertEquals("Unexpected exception status code", status, e.code());
-      assertTrue(e.getMessage().contains(getBaseUrl()));
-    }
-  }
-
   protected enum WT {
     JAVABIN,
     XML
diff --git a/solr/solrj/src/test/org/apache/solr/client/solrj/impl/LBHttp2SolrClientTest.java b/solr/solrj/src/test/org/apache/solr/client/solrj/impl/LBHttp2SolrClientTest.java
index 5f55b353c0d..3a93488e726 100644
--- a/solr/solrj/src/test/org/apache/solr/client/solrj/impl/LBHttp2SolrClientTest.java
+++ b/solr/solrj/src/test/org/apache/solr/client/solrj/impl/LBHttp2SolrClientTest.java
@@ -16,6 +16,7 @@
  */
 package org.apache.solr.client.solrj.impl;
 
+import java.io.IOException;
 import java.util.ArrayList;
 import java.util.Collections;
 import java.util.HashSet;
@@ -27,7 +28,9 @@
 import java.util.concurrent.TimeUnit;
 import java.util.concurrent.TimeoutException;
 import org.apache.solr.SolrTestCase;
+import org.apache.solr.client.solrj.SolrClientFunction;
 import org.apache.solr.client.solrj.SolrRequest;
+import org.apache.solr.client.solrj.SolrServerException;
 import org.apache.solr.client.solrj.request.QueryRequest;
 import org.apache.solr.common.SolrException;
 import org.apache.solr.common.params.MapSolrParams;
@@ -163,9 +166,11 @@ public void testAsync() {
         int index = Integer.parseInt(lastQueryReq.getParams().get("q"));
         assertNull("Found same request twice: " + index, queryRequests[index]);
         queryRequests[index] = lastQueryReq;
-        if (lastQueryReq.getBasePath().equals(ep1.toString())) {
+
+        final String lastBasePath = client.lastBasePaths.get(i);
+        if (lastBasePath.equals(ep1.toString())) {
           numEndpointOne++;
-        } else if (lastQueryReq.getBasePath().equals(ep2.toString())) {
+        } else if (lastBasePath.equals(ep2.toString())) {
           numEndpointTwo++;
         }
 
@@ -205,6 +210,8 @@ public static class MockHttp2SolrClient extends Http2SolrClient {
 
     public String basePathToFail = null;
 
+    public String tmpBaseUrl = null;
+
     protected MockHttp2SolrClient(String serverBaseUrl, Builder builder) {
       // TODO: Consider creating an interface for Http*SolrClient
       // so mocks can Implement, not Extend, and not actually need to
@@ -212,14 +219,27 @@ protected MockHttp2SolrClient(String serverBaseUrl, Builder builder) {
       super(serverBaseUrl, builder);
     }
 
+    @Override
+    public <R> R requestWithBaseUrl(
+        String baseUrl, SolrClientFunction<Http2SolrClient, R> clientFunction)
+        throws SolrServerException, IOException {
+      // This use of 'tmpBaseUrl' is NOT thread safe, but that's fine for our purposes here.
+      try {
+        tmpBaseUrl = baseUrl;
+        return clientFunction.apply(this);
+      } finally {
+        tmpBaseUrl = null;
+      }
+    }
+
     @Override
     public CompletableFuture<NamedList<Object>> requestAsync(
         final SolrRequest<?> solrRequest, String collection) {
       CompletableFuture<NamedList<Object>> cf = new CompletableFuture<>();
       lastSolrRequests.add(solrRequest);
-      lastBasePaths.add(solrRequest.getBasePath());
+      lastBasePaths.add(tmpBaseUrl);
       lastCollections.add(collection);
-      if (solrRequest.getBasePath().equals(basePathToFail)) {
+      if (tmpBaseUrl != null && tmpBaseUrl.equals(basePathToFail)) {
         cf.completeExceptionally(
             new SolrException(SolrException.ErrorCode.SERVER_ERROR, "We should retry this."));
       } else {

From 783d7805ea7ef6defcb87cdec55a662935a938fb Mon Sep 17 00:00:00 2001
From: Sanjay Dutt <sanjaydutt@apache.org>
Date: Tue, 5 Nov 2024 20:14:24 +0530
Subject: [PATCH 25/32] SOLR-17538: CloudHttp2SolrClient needs a custom
 ClusterStateProvider option (#2832)

* SOLR-17538: CloudHttp2SolrClient needs a custom ClusterStateProvider option

---------

Co-authored-by: David Smiley <dsmiley@apache.org>
---
 .../org/apache/solr/cloud/ZkController.java   |  3 +-
 .../org/apache/solr/cloud/OverseerTest.java   |  8 ++---
 .../solrj/impl/CloudHttp2SolrClient.java      | 35 ++++++++++++++-----
 .../client/solrj/impl/CloudSolrClient.java    |  5 +++
 .../impl/CloudHttp2SolrClientBuilderTest.java | 24 ++++++++-----
 5 files changed, 54 insertions(+), 21 deletions(-)

diff --git a/solr/core/src/java/org/apache/solr/cloud/ZkController.java b/solr/core/src/java/org/apache/solr/cloud/ZkController.java
index 25c5cb1fa9e..59480caa8e9 100644
--- a/solr/core/src/java/org/apache/solr/cloud/ZkController.java
+++ b/solr/core/src/java/org/apache/solr/cloud/ZkController.java
@@ -60,6 +60,7 @@
 import org.apache.solr.client.solrj.impl.HttpSolrClient.Builder;
 import org.apache.solr.client.solrj.impl.SolrClientCloudManager;
 import org.apache.solr.client.solrj.impl.SolrZkClientTimeout;
+import org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider;
 import org.apache.solr.client.solrj.request.CoreAdminRequest.WaitForState;
 import org.apache.solr.cloud.overseer.ClusterStateMutator;
 import org.apache.solr.cloud.overseer.OverseerAction;
@@ -870,7 +871,7 @@ public SolrCloudManager getSolrCloudManager() {
               .withConnectionTimeout(15000, TimeUnit.MILLISECONDS)
               .build();
       cloudSolrClient =
-          new CloudHttp2SolrClient.Builder(List.of(getZkServerAddress()), Optional.empty())
+          new CloudHttp2SolrClient.Builder(new ZkClientClusterStateProvider(zkStateReader))
               .withHttpClient(http2SolrClient)
               .build();
       cloudManager = new SolrClientCloudManager(cloudSolrClient, cc.getObjectCache());
diff --git a/solr/core/src/test/org/apache/solr/cloud/OverseerTest.java b/solr/core/src/test/org/apache/solr/cloud/OverseerTest.java
index ae10c722b20..8e7e45333f4 100644
--- a/solr/core/src/test/org/apache/solr/cloud/OverseerTest.java
+++ b/solr/core/src/test/org/apache/solr/cloud/OverseerTest.java
@@ -41,7 +41,6 @@
 import java.util.List;
 import java.util.Locale;
 import java.util.Map;
-import java.util.Optional;
 import java.util.Set;
 import java.util.concurrent.ExecutorService;
 import java.util.concurrent.TimeUnit;
@@ -55,6 +54,7 @@
 import org.apache.solr.client.solrj.impl.CloudHttp2SolrClient;
 import org.apache.solr.client.solrj.impl.Http2SolrClient;
 import org.apache.solr.client.solrj.impl.SolrClientCloudManager;
+import org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider;
 import org.apache.solr.cloud.overseer.NodeMutator;
 import org.apache.solr.cloud.overseer.OverseerAction;
 import org.apache.solr.cloud.overseer.ZkWriteCommand;
@@ -1942,18 +1942,18 @@ public Void answer(InvocationOnMock invocation) {
     when(zkController.getLeaderProps(anyString(), anyString(), anyInt())).thenCallRealMethod();
     when(zkController.getLeaderProps(anyString(), anyString(), anyInt(), anyBoolean()))
         .thenCallRealMethod();
-    doReturn(getCloudDataProvider(zkAddress)).when(zkController).getSolrCloudManager();
+    doReturn(getCloudDataProvider(reader)).when(zkController).getSolrCloudManager();
     return zkController;
   }
 
-  private SolrCloudManager getCloudDataProvider(String zkAddress) {
+  private SolrCloudManager getCloudDataProvider(ZkStateReader zkStateReader) {
     var httpSolrClient =
         new Http2SolrClient.Builder()
             .withIdleTimeout(30000, TimeUnit.MILLISECONDS)
             .withConnectionTimeout(15000, TimeUnit.MILLISECONDS)
             .build();
     var cloudSolrClient =
-        new CloudHttp2SolrClient.Builder(Collections.singletonList(zkAddress), Optional.empty())
+        new CloudHttp2SolrClient.Builder(new ZkClientClusterStateProvider(zkStateReader))
             .withHttpClient(httpSolrClient)
             .build();
     solrClients.add(cloudSolrClient);
diff --git a/solr/solrj/src/java/org/apache/solr/client/solrj/impl/CloudHttp2SolrClient.java b/solr/solrj/src/java/org/apache/solr/client/solrj/impl/CloudHttp2SolrClient.java
index f45945ae045..92da218d15c 100644
--- a/solr/solrj/src/java/org/apache/solr/client/solrj/impl/CloudHttp2SolrClient.java
+++ b/solr/solrj/src/java/org/apache/solr/client/solrj/impl/CloudHttp2SolrClient.java
@@ -56,10 +56,7 @@ protected CloudHttp2SolrClient(Builder builder) {
     super(builder.shardLeadersOnly, builder.parallelUpdates, builder.directUpdatesToLeadersOnly);
     this.clientIsInternal = builder.httpClient == null;
     this.myClient = createOrGetHttpClientFromBuilder(builder);
-    this.stateProvider =
-        builder.zkHosts.isEmpty()
-            ? createHttp2ClusterStateProvider(builder.solrUrls, myClient)
-            : createZkClusterStateProvider(builder);
+    this.stateProvider = createClusterStateProvider(builder);
     this.retryExpiryTimeNano = builder.retryExpiryTimeNano;
     this.defaultCollection = builder.defaultCollection;
     if (builder.requestWriter != null) {
@@ -89,6 +86,16 @@ private Http2SolrClient createOrGetHttpClientFromBuilder(Builder builder) {
     }
   }
 
+  private ClusterStateProvider createClusterStateProvider(Builder builder) {
+    if (builder.stateProvider != null) {
+      return builder.stateProvider;
+    } else if (builder.zkHosts.isEmpty()) {
+      return createHttp2ClusterStateProvider(builder.solrUrls, this.myClient);
+    } else {
+      return createZkClusterStateProvider(builder);
+    }
+  }
+
   private ClusterStateProvider createZkClusterStateProvider(Builder builder) {
     try {
       ClusterStateProvider stateProvider =
@@ -233,6 +240,11 @@ public Builder(List<String> zkHosts, Optional<String> zkChroot) {
       if (zkChroot.isPresent()) this.zkChroot = zkChroot.get();
     }
 
+    /** for an expert use-case */
+    public Builder(ClusterStateProvider stateProvider) {
+      this.stateProvider = stateProvider;
+    }
+
     /** Whether to use the default ZK ACLs when building a ZK Client. */
     public Builder canUseZkACLs(boolean canUseZkACLs) {
       this.canUseZkACLs = canUseZkACLs;
@@ -451,12 +463,19 @@ public Builder withZkClientTimeout(int zkClientTimeout, TimeUnit unit) {
 
     /** Create a {@link CloudHttp2SolrClient} based on the provided configuration. */
     public CloudHttp2SolrClient build() {
-      if (!zkHosts.isEmpty() && !solrUrls.isEmpty()) {
+      int providedOptions = 0;
+      if (!zkHosts.isEmpty()) providedOptions++;
+      if (!solrUrls.isEmpty()) providedOptions++;
+      if (stateProvider != null) providedOptions++;
+
+      if (providedOptions > 1) {
         throw new IllegalArgumentException(
-            "Both zkHost(s) & solrUrl(s) have been specified. Only specify one.");
-      } else if (zkHosts.isEmpty() && solrUrls.isEmpty()) {
-        throw new IllegalArgumentException("Both zkHosts and solrUrl cannot be null.");
+            "Only one of zkHost(s), solrUrl(s), or stateProvider should be specified.");
+      } else if (providedOptions == 0) {
+        throw new IllegalArgumentException(
+            "One of zkHosts, solrUrls, or stateProvider must be specified.");
       }
+
       return new CloudHttp2SolrClient(this);
     }
   }
diff --git a/solr/solrj/src/java/org/apache/solr/client/solrj/impl/CloudSolrClient.java b/solr/solrj/src/java/org/apache/solr/client/solrj/impl/CloudSolrClient.java
index 666f5662627..38cd77ddc4e 100644
--- a/solr/solrj/src/java/org/apache/solr/client/solrj/impl/CloudSolrClient.java
+++ b/solr/solrj/src/java/org/apache/solr/client/solrj/impl/CloudSolrClient.java
@@ -177,6 +177,11 @@ public Builder(List<String> solrUrls) {
     public Builder(List<String> zkHosts, Optional<String> zkChroot) {
       super(zkHosts, zkChroot);
     }
+
+    /** for an expert use-case */
+    public Builder(ClusterStateProvider stateProvider) {
+      super(stateProvider);
+    }
   }
 
   static class StateCache extends ConcurrentHashMap<String, ExpiringCachedDocCollection> {
diff --git a/solr/solrj/src/test/org/apache/solr/client/solrj/impl/CloudHttp2SolrClientBuilderTest.java b/solr/solrj/src/test/org/apache/solr/client/solrj/impl/CloudHttp2SolrClientBuilderTest.java
index 19c32d9c138..01f57970e99 100644
--- a/solr/solrj/src/test/org/apache/solr/client/solrj/impl/CloudHttp2SolrClientBuilderTest.java
+++ b/solr/solrj/src/test/org/apache/solr/client/solrj/impl/CloudHttp2SolrClientBuilderTest.java
@@ -17,6 +17,7 @@
 
 package org.apache.solr.client.solrj.impl;
 
+import static org.mockito.Mockito.mock;
 import static org.mockito.Mockito.never;
 import static org.mockito.Mockito.times;
 import static org.mockito.Mockito.verify;
@@ -30,7 +31,6 @@
 import org.apache.solr.cloud.SolrCloudTestCase;
 import org.junit.BeforeClass;
 import org.junit.Test;
-import org.mockito.Mockito;
 
 public class CloudHttp2SolrClientBuilderTest extends SolrCloudTestCase {
   private static final String ANY_CHROOT = "/ANY_CHROOT";
@@ -128,23 +128,23 @@ public void testExternalClientAndInternalBuilderTogether() {
         () ->
             new CloudHttp2SolrClient.Builder(
                     Collections.singletonList(ANY_ZK_HOST), Optional.of(ANY_CHROOT))
-                .withHttpClient(Mockito.mock(Http2SolrClient.class))
-                .withInternalClientBuilder(Mockito.mock(Http2SolrClient.Builder.class))
+                .withHttpClient(mock(Http2SolrClient.class))
+                .withInternalClientBuilder(mock(Http2SolrClient.Builder.class))
                 .build());
     expectThrows(
         IllegalStateException.class,
         () ->
             new CloudHttp2SolrClient.Builder(
                     Collections.singletonList(ANY_ZK_HOST), Optional.of(ANY_CHROOT))
-                .withInternalClientBuilder(Mockito.mock(Http2SolrClient.Builder.class))
-                .withHttpClient(Mockito.mock(Http2SolrClient.class))
+                .withInternalClientBuilder(mock(Http2SolrClient.Builder.class))
+                .withHttpClient(mock(Http2SolrClient.class))
                 .build());
   }
 
   @Test
   public void testProvideInternalBuilder() throws IOException {
-    Http2SolrClient http2Client = Mockito.mock(Http2SolrClient.class);
-    Http2SolrClient.Builder http2ClientBuilder = Mockito.mock(Http2SolrClient.Builder.class);
+    Http2SolrClient http2Client = mock(Http2SolrClient.class);
+    Http2SolrClient.Builder http2ClientBuilder = mock(Http2SolrClient.Builder.class);
     when(http2ClientBuilder.build()).thenReturn(http2Client);
     CloudHttp2SolrClient.Builder clientBuilder =
         new CloudHttp2SolrClient.Builder(
@@ -162,7 +162,7 @@ public void testProvideInternalBuilder() throws IOException {
 
   @Test
   public void testProvideExternalClient() throws IOException {
-    Http2SolrClient http2Client = Mockito.mock(Http2SolrClient.class);
+    Http2SolrClient http2Client = mock(Http2SolrClient.class);
     CloudHttp2SolrClient.Builder clientBuilder =
         new CloudHttp2SolrClient.Builder(
                 Collections.singletonList(ANY_ZK_HOST), Optional.of(ANY_CHROOT))
@@ -228,4 +228,12 @@ private void testHttpClientConsistency(
           ((Http2ClusterStateProvider) client.getClusterStateProvider()).getHttpClient());
     }
   }
+
+  public void testCustomStateProvider() throws IOException {
+    ZkClientClusterStateProvider stateProvider = mock(ZkClientClusterStateProvider.class);
+    try (CloudSolrClient cloudSolrClient = new CloudSolrClient.Builder(stateProvider).build()) {
+      assertEquals(stateProvider, cloudSolrClient.getClusterStateProvider());
+    }
+    verify(stateProvider, times(1)).close();
+  }
 }

From aaa2401040bbb7ce884c7eba91a496884eb1252a Mon Sep 17 00:00:00 2001
From: Houston Putman <houston@apache.org>
Date: Tue, 5 Nov 2024 14:27:09 -0600
Subject: [PATCH 26/32] SOLR-17545: Upgrade to Gradle 8.10 (#2848)

---
 .../lucene/gradle/WrapperDownloader.java      |  4 ++--
 gradle/documentation/render-javadoc.gradle    |  2 +-
 gradle/globals.gradle                         |  2 +-
 gradle/node.gradle                            |  2 +-
 gradle/solr/packaging.gradle                  |  2 +-
 gradle/testing/alternative-jdk-support.gradle |  2 +-
 gradle/testing/beasting.gradle                |  2 +-
 gradle/testing/profiling.gradle               |  2 +-
 gradle/testing/randomization.gradle           |  8 +++----
 gradle/testing/slowest-tests-at-end.gradle    |  2 +-
 gradle/validation/check-environment.gradle    |  2 +-
 gradle/validation/jar-checks.gradle           | 22 ++++++++++++-----
 gradle/validation/spotless.gradle             |  2 +-
 gradle/wrapper/gradle-wrapper.jar.sha256      |  2 +-
 gradle/wrapper/gradle-wrapper.jar.version     |  2 +-
 gradle/wrapper/gradle-wrapper.properties      |  3 ++-
 gradlew                                       |  9 ++++---
 gradlew.bat                                   | 24 ++++++++++---------
 settings.gradle                               |  2 +-
 19 files changed, 56 insertions(+), 40 deletions(-)

diff --git a/build-tools/build-infra/src/main/java/org/apache/lucene/gradle/WrapperDownloader.java b/build-tools/build-infra/src/main/java/org/apache/lucene/gradle/WrapperDownloader.java
index f7e07eb164a..88d91028552 100644
--- a/build-tools/build-infra/src/main/java/org/apache/lucene/gradle/WrapperDownloader.java
+++ b/build-tools/build-infra/src/main/java/org/apache/lucene/gradle/WrapperDownloader.java
@@ -55,9 +55,9 @@ public static void main(String[] args) {
 
   public static void checkVersion() {
     int major = Runtime.getRuntime().version().feature();
-    if (major < 11 || major > 21) {
+    if (major < 21 || major > 23) {
       throw new IllegalStateException(
-          "java version must be between 11 and 21, your version: " + major);
+          "java version must be between 21 and 23, your version: " + major);
     }
   }
 
diff --git a/gradle/documentation/render-javadoc.gradle b/gradle/documentation/render-javadoc.gradle
index bd90ad35426..6c637e540df 100644
--- a/gradle/documentation/render-javadoc.gradle
+++ b/gradle/documentation/render-javadoc.gradle
@@ -32,7 +32,7 @@ allprojects {
       missingdoclet "org.apache.solr.tools:missing-doclet"
     }
 
-    ext {
+    project.ext {
       relativeDocPath = project.path.replaceFirst(/:\w+:/, "").replace(':', '/')
     }
 
diff --git a/gradle/globals.gradle b/gradle/globals.gradle
index 4f782ae38c5..d8a99de69c2 100644
--- a/gradle/globals.gradle
+++ b/gradle/globals.gradle
@@ -37,7 +37,7 @@ allprojects {
   // so :solr:core will have solr-core.jar, etc.
   project.archivesBaseName = project.path.replaceAll("^:", "").replace(':', '-')
 
-  ext {
+  project.ext {
     // Utility method to support passing overrides via -P or -D.
     propertyOrDefault = { propName, defValue ->
       def result
diff --git a/gradle/node.gradle b/gradle/node.gradle
index 3da3a51d40e..ff8404c2db2 100644
--- a/gradle/node.gradle
+++ b/gradle/node.gradle
@@ -34,7 +34,7 @@ configure([project(":solr:packaging"), project(":solr:solr-ref-guide"), project(
         }
     }
 
-    ext {
+    project.ext {
         rootNodeDir = "$rootDir/.gradle/node"
         nodeProjectDir = file("$rootNodeDir/$project.name")
     }
diff --git a/gradle/solr/packaging.gradle b/gradle/solr/packaging.gradle
index 11ffb560688..1b5325f908b 100644
--- a/gradle/solr/packaging.gradle
+++ b/gradle/solr/packaging.gradle
@@ -38,7 +38,7 @@
 
 configure(allprojects.findAll {project -> project.path.startsWith(":solr:modules:") || project.path == ":solr:prometheus-exporter" || project.path == ":solr:cross-dc-manager" }) {
   plugins.withType(JavaPlugin) {
-    ext {
+    project.ext {
       packagingDir = file("${buildDir}/packaging")
       if (project.path.startsWith(":solr:prometheus-exporter") || project.path.startsWith(":solr:cross-dc-manager")) {
         deps = packagingDir
diff --git a/gradle/testing/alternative-jdk-support.gradle b/gradle/testing/alternative-jdk-support.gradle
index 72cdabdab4b..a1ff1b4b014 100644
--- a/gradle/testing/alternative-jdk-support.gradle
+++ b/gradle/testing/alternative-jdk-support.gradle
@@ -87,6 +87,6 @@ if (jvmGradle != jvmCurrent) {
 
 // Set up root project's properties.
 rootProject.ext.runtimeJavaHome = jvmCurrent.javaHome
-rootProject.ext.runtimeJavaVersion = jvmDetector.getMetadata(new InstallationLocation(jvmCurrent.javaHome, "specific path")).getLanguageVersion()
+rootProject.ext.runtimeJavaVersion = jvmDetector.getMetadata(InstallationLocation.userDefined(jvmCurrent.javaHome, "specific path")).getLanguageVersion()
 rootProject.ext.usesAltJvm = (jvmGradle != jvmCurrent);
 
diff --git a/gradle/testing/beasting.gradle b/gradle/testing/beasting.gradle
index 8934100ec10..67c20140ba8 100644
--- a/gradle/testing/beasting.gradle
+++ b/gradle/testing/beasting.gradle
@@ -27,7 +27,7 @@ def beastingMode = gradle.startParameter.taskNames.any{ name -> name == 'beast'
 
 allprojects {
   plugins.withType(JavaPlugin) {
-    ext {
+    project.ext {
       testOptions += [
           [propName: 'tests.dups', value: 0, description: "Reiterate runs of entire test suites ('beast' task)."]
       ]
diff --git a/gradle/testing/profiling.gradle b/gradle/testing/profiling.gradle
index ce9e7d43e03..8b1e5147efc 100644
--- a/gradle/testing/profiling.gradle
+++ b/gradle/testing/profiling.gradle
@@ -19,7 +19,7 @@ def recordings = files()
 
 allprojects {
   plugins.withType(JavaPlugin) {
-    ext {
+    project.ext {
       testOptions += [
           [propName: 'tests.profile', value: false, description: "Enable java flight recorder profiling."]
       ]
diff --git a/gradle/testing/randomization.gradle b/gradle/testing/randomization.gradle
index 2b617b99aa8..183943a39e7 100644
--- a/gradle/testing/randomization.gradle
+++ b/gradle/testing/randomization.gradle
@@ -79,7 +79,7 @@ allprojects {
 // Configure test property defaults and their descriptions.
 allprojects {
   plugins.withType(JavaPlugin) {
-    ext {
+    project.ext {
       testOptions += [
           // seed, repetition and amplification.
           [propName: 'tests.seed', value: { -> rootSeed }, description: "Sets the master randomization seed."],
@@ -124,7 +124,7 @@ allprojects {
 // Add Solr-specific test configs settings.
 configure(allprojects.findAll {project -> project.path.startsWith(":solr") }) {
   plugins.withType(JavaPlugin) {
-    ext {
+    project.ext {
       testOptions += [
           [propName: 'tests.src.home', value: null, description: "See SOLR-14023."],
           [propName: 'solr.tests.use.numeric.points', value: null, description: "Point implementation to use (true=numerics, false=trie)."],
@@ -137,14 +137,14 @@ configure(allprojects.findAll {project -> project.path.startsWith(":solr") }) {
 allprojects {
   plugins.withType(JavaPlugin) {
     afterEvaluate {
-      ext.testOptionsResolved = testOptions.findAll { opt ->
+      project.ext.testOptionsResolved = testOptions.findAll { opt ->
         propertyOrDefault(opt.propName, opt.value) != null
       }.collectEntries { opt ->
         [(opt.propName): Objects.toString(resolvedTestOption(opt.propName))]
       }
 
       // Compute the "reproduce with" string.
-      ext.testOptionsForReproduceLine = testOptions.findAll { opt ->
+      project.ext.testOptionsForReproduceLine = testOptions.findAll { opt ->
         if (opt["includeInReproLine"] == false) {
           return false
         }
diff --git a/gradle/testing/slowest-tests-at-end.gradle b/gradle/testing/slowest-tests-at-end.gradle
index eaf9cd1a2f1..d24e523394d 100644
--- a/gradle/testing/slowest-tests-at-end.gradle
+++ b/gradle/testing/slowest-tests-at-end.gradle
@@ -22,7 +22,7 @@ def allSuites = []
 
 allprojects {
   plugins.withType(JavaPlugin) {
-    ext {
+    project.ext {
       testOptions += [
           [propName: 'tests.slowestTests', value: true, description: "Print the summary of the slowest tests."],
           [propName: 'tests.slowestSuites', value: true, description: "Print the summary of the slowest suites."]
diff --git a/gradle/validation/check-environment.gradle b/gradle/validation/check-environment.gradle
index 6fcaf761a19..6e5c01ef19c 100644
--- a/gradle/validation/check-environment.gradle
+++ b/gradle/validation/check-environment.gradle
@@ -22,7 +22,7 @@ import org.gradle.util.GradleVersion
 
 configure(rootProject) {
   ext {
-    expectedGradleVersion = '8.4'
+    expectedGradleVersion = '8.10'
   }
 
   wrapper {
diff --git a/gradle/validation/jar-checks.gradle b/gradle/validation/jar-checks.gradle
index 6c87c50f023..7547214299d 100644
--- a/gradle/validation/jar-checks.gradle
+++ b/gradle/validation/jar-checks.gradle
@@ -1,3 +1,5 @@
+import java.util.stream.Collectors
+
 /*
  * Licensed to the Apache Software Foundation (ASF) under one or more
  * contributor license agreements.  See the NOTICE file distributed with
@@ -74,14 +76,14 @@ subprojects {
   // Configure jarValidation configuration for all projects. Any dependency
   // declared on this configuration (or any configuration it extends from) will
   // be verified.
-  configurations {
+  project.configurations {
     jarValidation
   }
 
   // For Java projects, add all dependencies from the following configurations
   // to jar validation
   plugins.withType(JavaPlugin) {
-    configurations {
+    project.configurations {
       jarValidation {
         extendsFrom runtimeClasspath
         extendsFrom compileClasspath
@@ -107,16 +109,24 @@ subprojects {
       }
 
       def excludeRules = configurations.jarValidation.excludeRules
+      List<Map<String, String>> excludeRuleMaps;
+      if (excludeRules && excludeRules.size() > 0) {
+        excludeRuleMaps = excludeRules.stream().map {rule ->
+          if (rule.module != null) {
+            Map.of("group", rule.group, "module", rule.module)
+          } else {
+            Map.of("group", rule.group)
+          }
+        }.collect(Collectors.toList())
+      }
 
       ArrayDeque<ResolvedDependency> queue = new ArrayDeque<>()
       configurations.jarValidation.extendsFrom.each { conf ->
-        if (excludeRules) {
+        if (excludeRules && excludeRules.size() > 0) {
           conf = conf.copyRecursive()
           conf.canBeResolved = true
           conf.canBeConsumed = true
-          def newConfExcludeRules = new HashSet<>(conf.excludeRules)
-          newConfExcludeRules.addAll(excludeRules)
-          conf.excludeRules = newConfExcludeRules
+          excludeRuleMaps.forEach {conf.exclude(it)}
         }
         if (conf.canBeResolved) {
           queue.addAll(conf.resolvedConfiguration.firstLevelModuleDependencies)
diff --git a/gradle/validation/spotless.gradle b/gradle/validation/spotless.gradle
index 76fecdfad98..d4444a87671 100644
--- a/gradle/validation/spotless.gradle
+++ b/gradle/validation/spotless.gradle
@@ -24,7 +24,7 @@ configure(allprojects) { prj ->
   plugins.withType(JavaPlugin) {
     prj.apply plugin: 'com.diffplug.spotless'
 
-    ext {
+    project.ext {
       spotlessJavaSetup = (Action){
         it.toggleOffOn() // obviously, only to be used sparingly.
         // TODO: Work out how to support multiple different header files (we have
diff --git a/gradle/wrapper/gradle-wrapper.jar.sha256 b/gradle/wrapper/gradle-wrapper.jar.sha256
index f78f56fee75..67dead8f441 100644
--- a/gradle/wrapper/gradle-wrapper.jar.sha256
+++ b/gradle/wrapper/gradle-wrapper.jar.sha256
@@ -1 +1 @@
-0336f591bc0ec9aa0c9988929b93ecc916b3c1d52aed202c7381db144aa0ef15
+2db75c40782f5e8ba1fc278a5574bab070adccb2d21ca5a6e5ed840888448046
diff --git a/gradle/wrapper/gradle-wrapper.jar.version b/gradle/wrapper/gradle-wrapper.jar.version
index a2f28f43be3..dd78a707858 100644
--- a/gradle/wrapper/gradle-wrapper.jar.version
+++ b/gradle/wrapper/gradle-wrapper.jar.version
@@ -1 +1 @@
-8.4.0
+8.10.2
\ No newline at end of file
diff --git a/gradle/wrapper/gradle-wrapper.properties b/gradle/wrapper/gradle-wrapper.properties
index 744c64d1277..9355b415575 100644
--- a/gradle/wrapper/gradle-wrapper.properties
+++ b/gradle/wrapper/gradle-wrapper.properties
@@ -1,6 +1,7 @@
 distributionBase=GRADLE_USER_HOME
 distributionPath=wrapper/dists
-distributionUrl=https\://services.gradle.org/distributions/gradle-8.4-bin.zip
+distributionUrl=https\://services.gradle.org/distributions/gradle-8.10-bin.zip
 networkTimeout=10000
+validateDistributionUrl=true
 zipStoreBase=GRADLE_USER_HOME
 zipStorePath=wrapper/dists
diff --git a/gradlew b/gradlew
index 0aa671a76c2..c8ad2977471 100755
--- a/gradlew
+++ b/gradlew
@@ -15,6 +15,8 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 #
+# SPDX-License-Identifier: Apache-2.0
+#
 
 ##############################################################################
 #
@@ -55,7 +57,7 @@
 #       Darwin, MinGW, and NonStop.
 #
 #   (3) This script is generated from the Groovy template
-#       https://github.com/gradle/gradle/blob/HEAD/subprojects/plugins/src/main/resources/org/gradle/api/internal/plugins/unixStartScript.txt
+#       https://github.com/gradle/gradle/blob/HEAD/platforms/jvm/plugins-application/src/main/resources/org/gradle/api/internal/plugins/unixStartScript.txt
 #       within the Gradle project.
 #
 #       You can find Gradle at https://github.com/gradle/gradle/.
@@ -84,7 +86,8 @@ done
 # shellcheck disable=SC2034
 APP_BASE_NAME=${0##*/}
 # Discard cd standard output in case $CDPATH is set (https://github.com/gradle/gradle/issues/25036)
-APP_HOME=$( cd "${APP_HOME:-./}" > /dev/null && pwd -P ) || exit
+APP_HOME=$( cd -P "${APP_HOME:-./}" > /dev/null && printf '%s
+' "$PWD" ) || exit
 
 # Use the maximum available, or set MAX_FD != -1 to use that value.
 MAX_FD=maximum
@@ -161,7 +164,7 @@ if [ ! -e "$GRADLE_WRAPPER_JAR" ]; then
     "$JAVACMD" $JAVA_OPTS "$APP_HOME/build-tools/build-infra/src/main/java/org/apache/lucene/gradle/WrapperDownloader.java" "$GRADLE_WRAPPER_JAR"
     WRAPPER_STATUS=$?
     if [ "$WRAPPER_STATUS" -eq 1 ]; then
-        echo "ERROR: Something went wrong. Make sure you're using Java version between 11 and 21."
+        echo "ERROR: Something went wrong. Make sure you're using Java version between 21 and 23."
         exit $WRAPPER_STATUS
     elif [ "$WRAPPER_STATUS" -ne 0 ]; then
         exit $WRAPPER_STATUS
diff --git a/gradlew.bat b/gradlew.bat
index 938e3ce94ee..ff65d8f6012 100644
--- a/gradlew.bat
+++ b/gradlew.bat
@@ -13,6 +13,8 @@
 @rem See the License for the specific language governing permissions and
 @rem limitations under the License.
 @rem
+@rem SPDX-License-Identifier: Apache-2.0
+@rem
 
 @if "%DEBUG%"=="" @echo off
 @rem ##########################################################################
@@ -48,11 +50,11 @@ set JAVA_EXE=java.exe
 %JAVA_EXE% -version >NUL 2>&1
 if %ERRORLEVEL% equ 0 goto execute
 
-echo.
-echo ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH.
-echo.
-echo Please set the JAVA_HOME variable in your environment to match the
-echo location of your Java installation.
+echo. 1>&2
+echo ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH. 1>&2
+echo. 1>&2
+echo Please set the JAVA_HOME variable in your environment to match the 1>&2
+echo location of your Java installation. 1>&2
 
 goto fail
 
@@ -62,11 +64,11 @@ set JAVA_EXE=%JAVA_HOME%/bin/java.exe
 
 if exist "%JAVA_EXE%" goto execute
 
-echo.
-echo ERROR: JAVA_HOME is set to an invalid directory: %JAVA_HOME%
-echo.
-echo Please set the JAVA_HOME variable in your environment to match the
-echo location of your Java installation.
+echo. 1>&2
+echo ERROR: JAVA_HOME is set to an invalid directory: %JAVA_HOME% 1>&2
+echo. 1>&2
+echo Please set the JAVA_HOME variable in your environment to match the 1>&2
+echo location of your Java installation. 1>&2
 
 goto fail
 
@@ -108,7 +110,7 @@ goto fail
 
 :failWithJvmMessage
 @rem https://github.com/apache/lucene/pull/819
-echo Error: Something went wrong. Make sure you're using Java version between 11 and 21.
+echo Error: Something went wrong. Make sure you're using Java version between 21 and 23.
 
 :fail
 rem Set variable GRADLE_EXIT_CONSOLE if you need the _script_ return code instead of
diff --git a/settings.gradle b/settings.gradle
index a9dd4a99ba1..7035b75bfdf 100644
--- a/settings.gradle
+++ b/settings.gradle
@@ -25,7 +25,7 @@ pluginManagement {
 }
 
 plugins {
-    id 'com.gradle.develocity' version '3.17.6'
+    id 'com.gradle.develocity' version '3.18.1'
     id 'com.gradle.common-custom-user-data-gradle-plugin' version '2.0.2'
 }
 

From 559dba90a5c05ccbb2bec4485d829062af64e95b Mon Sep 17 00:00:00 2001
From: Houston Putman <houston@apache.org>
Date: Tue, 5 Nov 2024 15:52:55 -0600
Subject: [PATCH 27/32] SOLR-17545: Changes.txt

---
 solr/CHANGES.txt | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/solr/CHANGES.txt b/solr/CHANGES.txt
index e8008ded0d1..751e19073fd 100644
--- a/solr/CHANGES.txt
+++ b/solr/CHANGES.txt
@@ -221,6 +221,8 @@ led to the suppression of exceptions. (Andrey Bozhko)
 
 * SOLR-17534: Introduce ClusterState.getCollectionNames, a convenience method (David Smiley)
 
+* SOLR-17545: Upgrade to Gradle 8.10 (Houston Putman)
+
 ==================  9.7.1 ==================
 Bug Fixes
 ---------------------

From 6561135f0b42ec0c79292c8e73c8ee7be1eb5d26 Mon Sep 17 00:00:00 2001
From: Christos Malliaridis <malliaridis@apache.org>
Date: Tue, 5 Nov 2024 23:40:04 +0100
Subject: [PATCH 28/32] SOLR-17321: Update ecj compiler versions to JDK 17
 (#2804)

* Increase ecj version, fix one issue found

---------

Co-authored-by: Houston Putman <houston@apache.org>
---
 build-tools/scriptDepVersions.gradle                        | 2 +-
 gradle/validation/ecj-lint/ecj.javadocs.prefs               | 6 +++---
 .../org/apache/solr/common/util/FastJavaBinDecoder.java     | 3 +--
 3 files changed, 5 insertions(+), 6 deletions(-)

diff --git a/build-tools/scriptDepVersions.gradle b/build-tools/scriptDepVersions.gradle
index 25413b6791a..329905331a9 100644
--- a/build-tools/scriptDepVersions.gradle
+++ b/build-tools/scriptDepVersions.gradle
@@ -25,7 +25,7 @@ ext {
       "min-solrj-java-version" : "17",
       "apache-rat": "0.15",
       "commons-codec": "1.16.0",
-      "ecj": "3.36.0",
+      "ecj": "3.39.0",
       "javacc": "7.0.12",
       "jgit": "6.7.0.202309050840-r",
       "flexmark": "0.64.8",
diff --git a/gradle/validation/ecj-lint/ecj.javadocs.prefs b/gradle/validation/ecj-lint/ecj.javadocs.prefs
index 975707055ff..74278547699 100644
--- a/gradle/validation/ecj-lint/ecj.javadocs.prefs
+++ b/gradle/validation/ecj-lint/ecj.javadocs.prefs
@@ -5,8 +5,8 @@ org.eclipse.jdt.core.compiler.annotation.nonnullbydefault=org.eclipse.jdt.annota
 org.eclipse.jdt.core.compiler.annotation.nonnullisdefault=disabled
 org.eclipse.jdt.core.compiler.annotation.nullable=org.eclipse.jdt.annotation.Nullable
 org.eclipse.jdt.core.compiler.annotation.nullanalysis=disabled
-org.eclipse.jdt.core.compiler.codegen.targetPlatform=11
-org.eclipse.jdt.core.compiler.compliance=11
+org.eclipse.jdt.core.compiler.codegen.targetPlatform=17
+org.eclipse.jdt.core.compiler.compliance=17
 org.eclipse.jdt.core.compiler.doc.comment.support=enabled
 org.eclipse.jdt.core.compiler.problem.annotationSuperInterface=error
 org.eclipse.jdt.core.compiler.problem.assertIdentifier=error
@@ -93,4 +93,4 @@ org.eclipse.jdt.core.compiler.problem.unusedParameterWhenOverridingConcrete=disa
 org.eclipse.jdt.core.compiler.problem.unusedPrivateMember=ignore
 org.eclipse.jdt.core.compiler.problem.unusedWarningToken=ignore
 org.eclipse.jdt.core.compiler.problem.varargsArgumentNeedCast=error
-org.eclipse.jdt.core.compiler.source=11
+org.eclipse.jdt.core.compiler.source=17
diff --git a/solr/solrj/src/java/org/apache/solr/common/util/FastJavaBinDecoder.java b/solr/solrj/src/java/org/apache/solr/common/util/FastJavaBinDecoder.java
index 06be49fc8de..37d82822cad 100644
--- a/solr/solrj/src/java/org/apache/solr/common/util/FastJavaBinDecoder.java
+++ b/solr/solrj/src/java/org/apache/solr/common/util/FastJavaBinDecoder.java
@@ -44,7 +44,6 @@
 import static org.apache.solr.common.util.JavaBinCodec.SOLRDOC;
 import static org.apache.solr.common.util.JavaBinCodec.SOLRDOCLST;
 import static org.apache.solr.common.util.JavaBinCodec.SOLRINPUTDOC;
-import static org.apache.solr.common.util.JavaBinCodec.STR;
 import static org.apache.solr.common.util.JavaBinCodec.TAG_AND_LEN;
 import static org.apache.solr.common.util.JavaBinCodec.readVInt;
 
@@ -586,7 +585,7 @@ public void lazyRead(EntryImpl entry, StreamCodec codec) throws IOException {
     },
     // types that combine tag + length (or other info) in a single byte
     _TAG_AND_LEN(TAG_AND_LEN, UPPER_3_BITS, null),
-    _STR(STR, UPPER_3_BITS, DataEntry.Type.STR) {
+    _STR(JavaBinCodec.STR, UPPER_3_BITS, DataEntry.Type.STR) {
       @Override
       public void lazyRead(EntryImpl entry, StreamCodec codec) throws IOException {
         entry.size = readObjSz(codec, this);

From 01b38b35171d4b9b3395fd7b3ea4ba9505e7bba3 Mon Sep 17 00:00:00 2001
From: Eric Pugh <epugh@opensourceconnections.com>
Date: Wed, 6 Nov 2024 15:49:22 -0500
Subject: [PATCH 29/32] SOLR-17544: Prevent conflicting connection flags from
 being used via OptionGroup (#2725)

Specifically this is the -s and -z parameters.

---------

Signed-off-by: Eric Pugh <epugh@opensourceconnections.com>
Co-authored-by: Christos Malliaridis <c.malliaridis@gmail.com>
---
 solr/CHANGES.txt                              |   2 +
 .../src/java/org/apache/solr/cli/ApiTool.java |  30 ++-
 .../java/org/apache/solr/cli/AssertTool.java  | 238 +++++++++--------
 .../java/org/apache/solr/cli/AuthTool.java    | 235 ++++++++---------
 .../java/org/apache/solr/cli/ClusterTool.java |  45 ++--
 .../org/apache/solr/cli/CommonCLIOptions.java |  74 ++++++
 .../solr/cli/ConfigSetDownloadTool.java       |  50 ++--
 .../apache/solr/cli/ConfigSetUploadTool.java  |  50 ++--
 .../java/org/apache/solr/cli/ConfigTool.java  |  92 ++++---
 .../java/org/apache/solr/cli/CreateTool.java  | 145 ++++++-----
 .../java/org/apache/solr/cli/DeleteTool.java  |  79 +++---
 .../java/org/apache/solr/cli/ExportTool.java  | 128 ++++++----
 .../org/apache/solr/cli/HealthcheckTool.java  |  35 +--
 .../org/apache/solr/cli/LinkConfigTool.java   |  46 ++--
 .../java/org/apache/solr/cli/PackageTool.java | 118 +++++----
 .../org/apache/solr/cli/PostLogsTool.java     |  51 ++--
 .../java/org/apache/solr/cli/PostTool.java    | 228 +++++++++--------
 .../org/apache/solr/cli/RunExampleTool.java   | 240 ++++++++++--------
 .../apache/solr/cli/SnapshotCreateTool.java   |  51 ++--
 .../apache/solr/cli/SnapshotDeleteTool.java   |  51 ++--
 .../apache/solr/cli/SnapshotDescribeTool.java |  51 ++--
 .../apache/solr/cli/SnapshotExportTool.java   | 111 ++++----
 .../org/apache/solr/cli/SnapshotListTool.java |  32 +--
 .../src/java/org/apache/solr/cli/SolrCLI.java | 117 +--------
 .../java/org/apache/solr/cli/StatusTool.java  |  82 +++---
 .../src/java/org/apache/solr/cli/Tool.java    |  10 +-
 .../java/org/apache/solr/cli/ToolBase.java    |  33 ++-
 .../org/apache/solr/cli/UpdateACLTool.java    |   7 +-
 .../java/org/apache/solr/cli/VersionTool.java |   8 -
 .../java/org/apache/solr/cli/ZkCpTool.java    |  34 +--
 .../java/org/apache/solr/cli/ZkLsTool.java    |  17 +-
 .../org/apache/solr/cli/ZkMkrootTool.java     |  28 +-
 .../java/org/apache/solr/cli/ZkMvTool.java    |  10 +-
 .../java/org/apache/solr/cli/ZkRmTool.java    |  17 +-
 .../java/org/apache/solr/cli/ZkToolHelp.java  |  36 +--
 .../test/org/apache/solr/cli/SolrCLITest.java |   3 -
 solr/packaging/test/test_create.bats          |   5 +
 solr/packaging/test/test_status.bats          |   7 +-
 .../prometheus/exporter/SolrExporter.java     |   2 +-
 39 files changed, 1409 insertions(+), 1189 deletions(-)
 create mode 100644 solr/core/src/java/org/apache/solr/cli/CommonCLIOptions.java

diff --git a/solr/CHANGES.txt b/solr/CHANGES.txt
index 751e19073fd..3a03a54985a 100644
--- a/solr/CHANGES.txt
+++ b/solr/CHANGES.txt
@@ -26,6 +26,8 @@ Improvements
 * SOLR-16116: Apache Curator is now used to manage all Solr Zookeeper interactions. This should provide more stability in the Solr-Zookeeper interactions.
   The solrj-zookeeper module, now has a dependency on curator. (Houston Putman, Kevin Risden, Mike Drob, David Smiley)
 
+* SOLR-17544: Solr CLI will now stop when you combine mutually exclusive options.  Combining -s and -z options is a common example.  (Eric Pugh, Christos Malliaridis)
+
 Optimizations
 ---------------------
 (No changes)
diff --git a/solr/core/src/java/org/apache/solr/cli/ApiTool.java b/solr/core/src/java/org/apache/solr/cli/ApiTool.java
index e9af6116fa9..02f893b3753 100644
--- a/solr/core/src/java/org/apache/solr/cli/ApiTool.java
+++ b/solr/core/src/java/org/apache/solr/cli/ApiTool.java
@@ -19,9 +19,9 @@
 
 import java.io.PrintStream;
 import java.net.URI;
-import java.util.List;
 import org.apache.commons.cli.CommandLine;
 import org.apache.commons.cli.Option;
+import org.apache.commons.cli.Options;
 import org.apache.solr.client.solrj.SolrRequest;
 import org.apache.solr.client.solrj.impl.JsonMapResponseParser;
 import org.apache.solr.client.solrj.request.GenericSolrRequest;
@@ -36,6 +36,16 @@
  * <p>Used to send an arbitrary HTTP request to a Solr API endpoint.
  */
 public class ApiTool extends ToolBase {
+
+  private static final Option SOLR_URL_OPTION =
+      Option.builder("s")
+          .longOpt("solr-url")
+          .hasArg()
+          .argName("URL")
+          .required()
+          .desc("Send a GET request to a Solr API endpoint.")
+          .build();
+
   public ApiTool() {
     this(CLIO.getOutStream());
   }
@@ -50,22 +60,16 @@ public String getName() {
   }
 
   @Override
-  public List<Option> getOptions() {
-    return List.of(
-        Option.builder()
-            .longOpt("solr-url")
-            .argName("URL")
-            .hasArg()
-            .required(true)
-            .desc("Send a GET request to a Solr API endpoint.")
-            .build(),
-        SolrCLI.OPTION_CREDENTIALS);
+  public Options getOptions() {
+    return super.getOptions()
+        .addOption(SOLR_URL_OPTION)
+        .addOption(CommonCLIOptions.CREDENTIALS_OPTION);
   }
 
   @Override
   public void runImpl(CommandLine cli) throws Exception {
-    String getUrl = cli.getOptionValue("solr-url");
-    String response = callGet(getUrl, cli.getOptionValue(SolrCLI.OPTION_CREDENTIALS.getLongOpt()));
+    String getUrl = cli.getOptionValue(SOLR_URL_OPTION);
+    String response = callGet(getUrl, cli.getOptionValue(CommonCLIOptions.CREDENTIALS_OPTION));
 
     // pretty-print the response to stdout
     echo(response);
diff --git a/solr/core/src/java/org/apache/solr/cli/AssertTool.java b/solr/core/src/java/org/apache/solr/cli/AssertTool.java
index 966e7993146..b710d0934b2 100644
--- a/solr/core/src/java/org/apache/solr/cli/AssertTool.java
+++ b/solr/core/src/java/org/apache/solr/cli/AssertTool.java
@@ -23,10 +23,11 @@
 import java.nio.file.Path;
 import java.nio.file.Paths;
 import java.nio.file.attribute.FileOwnerAttributeView;
-import java.util.List;
 import java.util.concurrent.TimeUnit;
 import org.apache.commons.cli.CommandLine;
 import org.apache.commons.cli.Option;
+import org.apache.commons.cli.OptionGroup;
+import org.apache.commons.cli.Options;
 import org.apache.solr.client.solrj.SolrClient;
 import org.apache.solr.client.solrj.SolrServerException;
 import org.apache.solr.client.solrj.request.HealthCheckRequest;
@@ -44,6 +45,105 @@ public class AssertTool extends ToolBase {
   private static boolean useExitCode = false;
   private static Long timeoutMs = 1000L;
 
+  private static final Option IS_NOT_ROOT_OPTION =
+      Option.builder().desc("Asserts that we are NOT the root user.").longOpt("not-root").build();
+
+  private static final Option IS_ROOT_OPTION =
+      Option.builder().desc("Asserts that we are the root user.").longOpt("root").build();
+
+  private static final OptionGroup ROOT_OPTION =
+      new OptionGroup().addOption(IS_NOT_ROOT_OPTION).addOption(IS_ROOT_OPTION);
+
+  private static final Option IS_NOT_RUNNING_ON_OPTION =
+      Option.builder()
+          .desc("Asserts that Solr is NOT running on a certain URL. Default timeout is 1000ms.")
+          .longOpt("not-started")
+          .hasArg()
+          .argName("url")
+          .build();
+
+  private static final Option IS_RUNNING_ON_OPTION =
+      Option.builder()
+          .desc("Asserts that Solr is running on a certain URL. Default timeout is 1000ms.")
+          .longOpt("started")
+          .hasArg()
+          .argName("url")
+          .build();
+
+  private static final OptionGroup RUNNING_OPTION =
+      new OptionGroup().addOption(IS_NOT_RUNNING_ON_OPTION).addOption(IS_RUNNING_ON_OPTION);
+
+  private static final Option SAME_USER_OPTION =
+      Option.builder()
+          .desc("Asserts that we run as same user that owns <directory>.")
+          .longOpt("same-user")
+          .hasArg()
+          .argName("directory")
+          .build();
+
+  private static final Option DIRECTORY_EXISTS_OPTION =
+      Option.builder()
+          .desc("Asserts that directory <directory> exists.")
+          .longOpt("exists")
+          .hasArg()
+          .argName("directory")
+          .build();
+
+  private static final Option DIRECTORY_NOT_EXISTS_OPTION =
+      Option.builder()
+          .desc("Asserts that directory <directory> does NOT exist.")
+          .longOpt("not-exists")
+          .hasArg()
+          .argName("directory")
+          .build();
+
+  private static final OptionGroup DIRECTORY_OPTION =
+      new OptionGroup().addOption(DIRECTORY_EXISTS_OPTION).addOption(DIRECTORY_NOT_EXISTS_OPTION);
+
+  private static final Option IS_CLOUD_OPTION =
+      Option.builder()
+          .desc(
+              "Asserts that Solr is running in cloud mode.  Also fails if Solr not running.  URL should be for root Solr path.")
+          .longOpt("cloud")
+          .hasArg()
+          .argName("url")
+          .build();
+
+  private static final Option IS_NOT_CLOUD_OPTION =
+      Option.builder()
+          .desc(
+              "Asserts that Solr is not running in cloud mode.  Also fails if Solr not running.  URL should be for root Solr path.")
+          .longOpt("not-cloud")
+          .hasArg()
+          .argName("url")
+          .build();
+
+  private static final OptionGroup CLOUD_OPTION =
+      new OptionGroup().addOption(IS_CLOUD_OPTION).addOption(IS_NOT_CLOUD_OPTION);
+
+  private static final Option MESSAGE_OPTION =
+      Option.builder()
+          .desc("Exception message to be used in place of the default error message.")
+          .longOpt("message")
+          .hasArg()
+          .argName("message")
+          .build();
+
+  private static final Option TIMEOUT_OPTION =
+      Option.builder()
+          .desc("Timeout in ms for commands supporting a timeout.")
+          .longOpt("timeout")
+          .hasArg()
+          .type(Long.class)
+          .argName("ms")
+          .build();
+
+  private static final Option EXIT_CODE_OPTION =
+      Option.builder()
+          .desc("Return an exit code instead of printing error message on assert fail.")
+          .longOpt("exitcode")
+          .build();
+
   public AssertTool() {
     this(CLIO.getOutStream());
   }
@@ -58,72 +158,17 @@ public String getName() {
   }
 
   @Override
-  public List<Option> getOptions() {
-    return List.of(
-        Option.builder().desc("Asserts that we are NOT the root user.").longOpt("not-root").build(),
-        Option.builder().desc("Asserts that we are the root user.").longOpt("root").build(),
-        Option.builder()
-            .desc("Asserts that Solr is NOT running on a certain URL. Default timeout is 1000ms.")
-            .longOpt("not-started")
-            .hasArg(true)
-            .argName("url")
-            .build(),
-        Option.builder()
-            .desc("Asserts that Solr is running on a certain URL. Default timeout is 1000ms.")
-            .longOpt("started")
-            .hasArg(true)
-            .argName("url")
-            .build(),
-        Option.builder()
-            .desc("Asserts that we run as same user that owns <directory>.")
-            .longOpt("same-user")
-            .hasArg(true)
-            .argName("directory")
-            .build(),
-        Option.builder()
-            .desc("Asserts that directory <directory> exists.")
-            .longOpt("exists")
-            .hasArg(true)
-            .argName("directory")
-            .build(),
-        Option.builder()
-            .desc("Asserts that directory <directory> does NOT exist.")
-            .longOpt("not-exists")
-            .hasArg(true)
-            .argName("directory")
-            .build(),
-        Option.builder()
-            .desc(
-                "Asserts that Solr is running in cloud mode.  Also fails if Solr not running.  URL should be for root Solr path.")
-            .longOpt("cloud")
-            .hasArg(true)
-            .argName("url")
-            .build(),
-        Option.builder()
-            .desc(
-                "Asserts that Solr is not running in cloud mode.  Also fails if Solr not running.  URL should be for root Solr path.")
-            .longOpt("not-cloud")
-            .hasArg(true)
-            .argName("url")
-            .build(),
-        Option.builder()
-            .desc("Exception message to be used in place of the default error message.")
-            .longOpt("message")
-            .hasArg(true)
-            .argName("message")
-            .build(),
-        Option.builder()
-            .desc("Timeout in ms for commands supporting a timeout.")
-            .longOpt("timeout")
-            .hasArg(true)
-            .type(Long.class)
-            .argName("ms")
-            .build(),
-        Option.builder()
-            .desc("Return an exit code instead of printing error message on assert fail.")
-            .longOpt("exitcode")
-            .build(),
-        SolrCLI.OPTION_CREDENTIALS);
+  public Options getOptions() {
+    return super.getOptions()
+        .addOptionGroup(ROOT_OPTION)
+        .addOptionGroup(RUNNING_OPTION)
+        .addOption(SAME_USER_OPTION)
+        .addOptionGroup(DIRECTORY_OPTION)
+        .addOptionGroup(CLOUD_OPTION)
+        .addOption(MESSAGE_OPTION)
+        .addOption(TIMEOUT_OPTION)
+        .addOption(EXIT_CODE_OPTION)
+        .addOption(CommonCLIOptions.CREDENTIALS_OPTION);
   }
 
   /**
@@ -137,8 +182,6 @@ public List<Option> getOptions() {
    */
   @Override
   public int runTool(CommandLine cli) throws Exception {
-    verbose = cli.hasOption(SolrCLI.OPTION_VERBOSE.getLongOpt());
-
     int toolExitStatus;
     try {
       toolExitStatus = runAssert(cli);
@@ -146,7 +189,7 @@ public int runTool(CommandLine cli) throws Exception {
       // since this is a CLI, spare the user the stacktrace
       String excMsg = exc.getMessage();
       if (excMsg != null) {
-        if (verbose) {
+        if (isVerbose()) {
           CLIO.err("\nERROR: " + exc + "\n");
         } else {
           CLIO.err("\nERROR: " + excMsg + "\n");
@@ -172,58 +215,49 @@ public void runImpl(CommandLine cli) throws Exception {
    * @throws Exception if a tool failed, e.g. authentication failure
    */
   protected int runAssert(CommandLine cli) throws Exception {
-    if (cli.hasOption("m")) {
-      message = cli.getOptionValue("m");
-    }
-    if (cli.hasOption("message")) {
-      message = cli.getOptionValue("message");
-    }
-    if (cli.hasOption("timeout")) {
-      timeoutMs = cli.getParsedOptionValue("timeout");
-    }
-    if (cli.hasOption("exitcode")) {
-      useExitCode = true;
-    }
+    message = cli.getOptionValue(MESSAGE_OPTION);
+    timeoutMs = cli.getParsedOptionValue(TIMEOUT_OPTION, timeoutMs);
+    useExitCode = cli.hasOption(EXIT_CODE_OPTION);
 
     int ret = 0;
-    if (cli.hasOption("root")) {
+    if (cli.hasOption(IS_ROOT_OPTION)) {
       ret += assertRootUser();
     }
-    if (cli.hasOption("not-root")) {
+    if (cli.hasOption(IS_NOT_ROOT_OPTION)) {
       ret += assertNotRootUser();
     }
-    if (cli.hasOption("exists")) {
-      ret += assertFileExists(cli.getOptionValue("exists"));
+    if (cli.hasOption(DIRECTORY_EXISTS_OPTION)) {
+      ret += assertFileExists(cli.getOptionValue(DIRECTORY_EXISTS_OPTION));
     }
-    if (cli.hasOption("not-exists")) {
-      ret += assertFileNotExists(cli.getOptionValue("not-exists"));
+    if (cli.hasOption(DIRECTORY_NOT_EXISTS_OPTION)) {
+      ret += assertFileNotExists(cli.getOptionValue(DIRECTORY_NOT_EXISTS_OPTION));
     }
-    if (cli.hasOption("same-user")) {
-      ret += sameUser(cli.getOptionValue("same-user"));
+    if (cli.hasOption(SAME_USER_OPTION)) {
+      ret += sameUser(cli.getOptionValue(SAME_USER_OPTION));
     }
-    if (cli.hasOption("started")) {
+    if (cli.hasOption(IS_RUNNING_ON_OPTION)) {
       ret +=
           assertSolrRunning(
-              cli.getOptionValue("started"),
-              cli.getOptionValue(SolrCLI.OPTION_CREDENTIALS.getLongOpt()));
+              cli.getOptionValue(IS_RUNNING_ON_OPTION),
+              cli.getOptionValue(CommonCLIOptions.CREDENTIALS_OPTION));
     }
-    if (cli.hasOption("not-started")) {
+    if (cli.hasOption(IS_NOT_RUNNING_ON_OPTION)) {
       ret +=
           assertSolrNotRunning(
-              cli.getOptionValue("not-started"),
-              cli.getOptionValue(SolrCLI.OPTION_CREDENTIALS.getLongOpt()));
+              cli.getOptionValue(IS_NOT_RUNNING_ON_OPTION),
+              cli.getOptionValue(CommonCLIOptions.CREDENTIALS_OPTION));
     }
-    if (cli.hasOption("cloud")) {
+    if (cli.hasOption(IS_CLOUD_OPTION)) {
       ret +=
           assertSolrRunningInCloudMode(
-              SolrCLI.normalizeSolrUrl(cli.getOptionValue("cloud")),
-              cli.getOptionValue(SolrCLI.OPTION_CREDENTIALS.getLongOpt()));
+              SolrCLI.normalizeSolrUrl(cli.getOptionValue(IS_CLOUD_OPTION)),
+              cli.getOptionValue(CommonCLIOptions.CREDENTIALS_OPTION));
     }
-    if (cli.hasOption("not-cloud")) {
+    if (cli.hasOption(IS_NOT_CLOUD_OPTION)) {
       ret +=
           assertSolrNotRunningInCloudMode(
-              SolrCLI.normalizeSolrUrl(cli.getOptionValue("not-cloud")),
-              cli.getOptionValue(SolrCLI.OPTION_CREDENTIALS.getLongOpt()));
+              SolrCLI.normalizeSolrUrl(cli.getOptionValue(IS_NOT_CLOUD_OPTION)),
+              cli.getOptionValue(CommonCLIOptions.CREDENTIALS_OPTION));
     }
     return ret;
   }
diff --git a/solr/core/src/java/org/apache/solr/cli/AuthTool.java b/solr/core/src/java/org/apache/solr/cli/AuthTool.java
index bc091073170..817b0093748 100644
--- a/solr/core/src/java/org/apache/solr/cli/AuthTool.java
+++ b/solr/core/src/java/org/apache/solr/cli/AuthTool.java
@@ -35,8 +35,8 @@
 import java.util.Locale;
 import java.util.stream.Collectors;
 import org.apache.commons.cli.CommandLine;
-import org.apache.commons.cli.HelpFormatter;
 import org.apache.commons.cli.Option;
+import org.apache.commons.cli.Options;
 import org.apache.lucene.util.Constants;
 import org.apache.solr.common.cloud.SolrZkClient;
 import org.apache.solr.common.util.StrUtils;
@@ -46,6 +46,70 @@
 
 /** Supports auth command in the bin/solr script. */
 public class AuthTool extends ToolBase {
+
+  private static final Option TYPE_OPTION =
+      Option.builder()
+          .longOpt("type")
+          .hasArg()
+          .desc(
+              "The authentication mechanism to enable (basicAuth or kerberos). Defaults to 'basicAuth'.")
+          .build();
+
+  private static final Option PROMPT_OPTION =
+      Option.builder()
+          .longOpt("prompt")
+          .hasArg()
+          .type(Boolean.class)
+          .desc(
+              "Prompts the user to provide the credentials. Use either --credentials or --prompt, not both.")
+          .build();
+
+  private static final Option CONFIG_OPTION =
+      Option.builder()
+          .longOpt("config")
+          .hasArgs()
+          .desc(
+              "Configuration parameters (Solr startup parameters). Required for Kerberos authentication.")
+          .build();
+
+  private static final Option BLOCK_UNKNOWN_OPTION =
+      Option.builder()
+          .longOpt("block-unknown")
+          .desc("Blocks all access for unknown users (requires authentication for all endpoints).")
+          .hasArg()
+          .argName("true|false")
+          .type(Boolean.class)
+          .build();
+
+  private static final Option SOLR_INCLUDE_FILE_OPTION =
+      Option.builder()
+          .longOpt("solr-include-file")
+          .hasArg()
+          .argName("FILE")
+          .desc(
+              "The Solr include file which contains overridable environment variables for configuring Solr configurations.")
+          .build();
+
+  private static final Option UPDATE_INCLUDE_FILE_OPTION =
+      Option.builder()
+          .longOpt("update-include-file-only")
+          .desc(
+              "Only update the solr.in.sh or solr.in.cmd file, and skip actual enabling/disabling"
+                  + " authentication (i.e. don't update security.json).")
+          .hasArg()
+          .type(Boolean.class)
+          .build();
+
+  private static final Option AUTH_CONF_DIR_OPTION =
+      Option.builder()
+          .longOpt("auth-conf-dir")
+          .hasArg()
+          .argName("FILE")
+          .required()
+          .desc(
+              "This is where any authentication related configuration files, if any, would be placed.")
+          .build();
+
   public AuthTool() {
     this(CLIO.getOutStream());
   }
@@ -84,79 +148,38 @@ public String getHeader() {
           "SOLR_AUTHENTICATION_CLIENT_BUILDER", "SOLR_AUTH_TYPE", "SOLR_AUTHENTICATION_OPTS");
 
   @Override
-  public List<Option> getOptions() {
-    return List.of(
-        Option.builder()
-            .longOpt("type")
-            .hasArg()
-            .desc(
-                "The authentication mechanism to enable (basicAuth or kerberos). Defaults to 'basicAuth'.")
-            .build(),
-        Option.builder()
-            .longOpt("prompt")
-            .hasArg()
-            .desc(
-                "Prompts the user to provide the credentials. Use either --credentials or --prompt, not both.")
-            .build(),
-        Option.builder()
-            .longOpt("config")
-            .hasArgs()
-            .desc(
-                "Configuration parameters (Solr startup parameters). Required for Kerberos authentication.")
-            .build(),
-        Option.builder()
-            .longOpt("block-unknown")
-            .desc(
-                "Blocks all access for unknown users (requires authentication for all endpoints).")
-            .hasArg()
-            .argName("true|false")
-            .build(),
-        Option.builder()
-            .longOpt("solr-include-file")
-            .hasArg()
-            .argName("FILE")
-            .desc(
-                "The Solr include file which contains overridable environment variables for configuring Solr configurations.")
-            .build(),
-        Option.builder()
-            .longOpt("update-include-file-only")
-            .desc(
-                "Only update the solr.in.sh or solr.in.cmd file, and skip actual enabling/disabling"
-                    + " authentication (i.e. don't update security.json).")
-            .hasArg()
-            .build(),
-        Option.builder()
-            .longOpt("auth-conf-dir")
-            .hasArg()
-            .argName("FILE")
-            .required()
-            .desc(
-                "This is where any authentication related configuration files, if any, would be placed.")
-            .build(),
-        SolrCLI.OPTION_SOLRURL,
-        SolrCLI.OPTION_ZKHOST,
-        SolrCLI.OPTION_CREDENTIALS);
+  public Options getOptions() {
+    return super.getOptions()
+        .addOption(TYPE_OPTION)
+        .addOption(PROMPT_OPTION)
+        .addOption(CONFIG_OPTION)
+        .addOption(BLOCK_UNKNOWN_OPTION)
+        .addOption(SOLR_INCLUDE_FILE_OPTION)
+        .addOption(UPDATE_INCLUDE_FILE_OPTION)
+        .addOption(AUTH_CONF_DIR_OPTION)
+        .addOption(CommonCLIOptions.CREDENTIALS_OPTION)
+        .addOptionGroup(getConnectionOptions());
   }
 
-  private void ensureArgumentIsValidBooleanIfPresent(CommandLine cli, String argName) {
-    if (cli.hasOption(argName)) {
-      final String value = cli.getOptionValue(argName);
+  private void ensureArgumentIsValidBooleanIfPresent(CommandLine cli, Option option) {
+    if (cli.hasOption(option)) {
+      final String value = cli.getOptionValue(option);
       if (!"true".equalsIgnoreCase(value) && !"false".equalsIgnoreCase(value)) {
-        echo("Argument [" + argName + "] must be either true or false, but was [" + value + "]");
+        echo(
+            "Argument ["
+                + option.getLongOpt()
+                + "] must be either true or false, but was ["
+                + value
+                + "]");
         SolrCLI.exit(1);
       }
     }
   }
 
-  // @Override
-  // public int runTool(CommandLine cli) throws Exception {
-
-  // }
-
-  private int handleKerberos(CommandLine cli) throws Exception {
+  private void handleKerberos(CommandLine cli) throws Exception {
     String cmd = cli.getArgs()[0];
     boolean updateIncludeFileOnly =
-        Boolean.parseBoolean(cli.getOptionValue("update-include-file-only", "false"));
+        Boolean.parseBoolean(cli.getOptionValue(UPDATE_INCLUDE_FILE_OPTION, "false"));
     String securityJson =
         "{"
             + "\n  \"authentication\":{"
@@ -176,7 +199,7 @@ private int handleKerberos(CommandLine cli) throws Exception {
             CLIO.out(
                 "Unable to access ZooKeeper. Please add the following security.json to ZooKeeper (in case of SolrCloud):\n"
                     + securityJson
-                    + "\n");
+                    + "Dsolr.httpclient.config=[basicAuthConfFile]\n");
             zkInaccessible = true;
           }
           if (zkHost == null) {
@@ -217,7 +240,7 @@ private int handleKerberos(CommandLine cli) throws Exception {
           }
         }
 
-        String config = StrUtils.join(Arrays.asList(cli.getOptionValues("config")), ' ');
+        String config = StrUtils.join(Arrays.asList(cli.getOptionValues(CONFIG_OPTION)), ' ');
         // config is base64 encoded (to get around parsing problems), decode it
         config = config.replace(" ", "");
         config =
@@ -226,7 +249,7 @@ private int handleKerberos(CommandLine cli) throws Exception {
                 StandardCharsets.UTF_8);
         config = config.replace("\n", "").replace("\r", "");
 
-        String solrIncludeFilename = cli.getOptionValue("solr-include-file");
+        String solrIncludeFilename = cli.getOptionValue(SOLR_INCLUDE_FILE_OPTION);
         File includeFile = new File(solrIncludeFilename);
         if (!includeFile.exists() || !includeFile.canWrite()) {
           CLIO.out(
@@ -236,15 +259,14 @@ private int handleKerberos(CommandLine cli) throws Exception {
         }
 
         // update the solr.in.sh file to contain the necessary authentication lines
-        updateIncludeFileEnableAuth(includeFile.toPath(), null, config, cli);
+        updateIncludeFileEnableAuth(includeFile.toPath(), null, config);
         echo(
             "Successfully enabled Kerberos authentication; please restart any running Solr nodes.");
-        return 0;
-
+        return;
       case "disable":
         clearSecurityJson(cli, updateIncludeFileOnly);
 
-        solrIncludeFilename = cli.getOptionValue("solr-include-file");
+        solrIncludeFilename = cli.getOptionValue(SOLR_INCLUDE_FILE_OPTION);
         includeFile = new File(solrIncludeFilename);
         if (!includeFile.exists() || !includeFile.canWrite()) {
           CLIO.out(
@@ -255,38 +277,31 @@ private int handleKerberos(CommandLine cli) throws Exception {
         }
 
         // update the solr.in.sh file to comment out the necessary authentication lines
-        updateIncludeFileDisableAuth(includeFile.toPath(), cli);
-        return 0;
-
+        updateIncludeFileDisableAuth(includeFile.toPath());
+        return;
       default:
         CLIO.out("Valid auth commands are: enable, disable.");
         SolrCLI.exit(1);
     }
 
     CLIO.out("Options not understood.");
-    new HelpFormatter()
-        .printHelp("bin/solr auth <enable|disable> [OPTIONS]", SolrCLI.getToolOptions(this));
-    return 1;
+    SolrCLI.exit(1);
   }
 
-  private int handleBasicAuth(CommandLine cli) throws Exception {
+  private void handleBasicAuth(CommandLine cli) throws Exception {
     String cmd = cli.getArgs()[0];
-    boolean prompt = Boolean.parseBoolean(cli.getOptionValue("prompt", "false"));
+    boolean prompt = Boolean.parseBoolean(cli.getOptionValue(PROMPT_OPTION, "false"));
     boolean updateIncludeFileOnly =
-        Boolean.parseBoolean(cli.getOptionValue("update-include-file-only", "false"));
+        Boolean.parseBoolean(cli.getOptionValue(UPDATE_INCLUDE_FILE_OPTION, "false"));
     switch (cmd) {
       case "enable":
-        if (!prompt && !cli.hasOption("credentials")) {
+        if (!prompt && !cli.hasOption(CommonCLIOptions.CREDENTIALS_OPTION)) {
           CLIO.out("Option --credentials or --prompt is required with enable.");
-          new HelpFormatter()
-              .printHelp("bin/solr auth <enable|disable> [OPTIONS]", SolrCLI.getToolOptions(this));
           SolrCLI.exit(1);
         } else if (!prompt
-            && (cli.getOptionValue(SolrCLI.OPTION_CREDENTIALS.getLongOpt()) == null
-                || !cli.getOptionValue(SolrCLI.OPTION_CREDENTIALS.getLongOpt()).contains(":"))) {
+            && (cli.getOptionValue(CommonCLIOptions.CREDENTIALS_OPTION) == null
+                || !cli.getOptionValue(CommonCLIOptions.CREDENTIALS_OPTION).contains(":"))) {
           CLIO.out("Option --credentials is not in correct format.");
-          new HelpFormatter()
-              .printHelp("bin/solr auth <enable|disable> [OPTIONS]", SolrCLI.getToolOptions(this));
           SolrCLI.exit(1);
         }
 
@@ -296,7 +311,7 @@ private int handleBasicAuth(CommandLine cli) throws Exception {
           try {
             zkHost = SolrCLI.getZkHost(cli);
           } catch (Exception ex) {
-            if (cli.hasOption("zk-host")) {
+            if (cli.hasOption(CommonCLIOptions.ZK_HOST_OPTION)) {
               CLIO.out(
                   "Couldn't get ZooKeeper host. Please make sure that ZooKeeper is running and the correct zk-host has been passed in.");
             } else {
@@ -306,7 +321,7 @@ private int handleBasicAuth(CommandLine cli) throws Exception {
             SolrCLI.exit(1);
           }
           if (zkHost == null) {
-            if (cli.hasOption("zk-host")) {
+            if (cli.hasOption(CommonCLIOptions.ZK_HOST_OPTION)) {
               CLIO.out(
                   "Couldn't get ZooKeeper host. Please make sure that ZooKeeper is running and the correct zk-host has been passed in.");
             } else {
@@ -323,8 +338,8 @@ private int handleBasicAuth(CommandLine cli) throws Exception {
         }
 
         String username, password;
-        if (cli.hasOption(SolrCLI.OPTION_CREDENTIALS.getLongOpt())) {
-          String credentials = cli.getOptionValue(SolrCLI.OPTION_CREDENTIALS.getLongOpt());
+        if (cli.hasOption(CommonCLIOptions.CREDENTIALS_OPTION)) {
+          String credentials = cli.getOptionValue(CommonCLIOptions.CREDENTIALS_OPTION);
           username = credentials.split(":")[0];
           password = credentials.split(":")[1];
         } else {
@@ -340,7 +355,8 @@ private int handleBasicAuth(CommandLine cli) throws Exception {
           } while (password.length() == 0);
         }
 
-        boolean blockUnknown = Boolean.parseBoolean(cli.getOptionValue("block-unknown", "true"));
+        boolean blockUnknown =
+            Boolean.parseBoolean(cli.getOptionValue(BLOCK_UNKNOWN_OPTION, "true"));
 
         String resourceName = "security.json";
         final URL resource = SolrCore.class.getClassLoader().getResource(resourceName);
@@ -370,7 +386,7 @@ private int handleBasicAuth(CommandLine cli) throws Exception {
           }
         }
 
-        String solrIncludeFilename = cli.getOptionValue("solr-include-file");
+        String solrIncludeFilename = cli.getOptionValue(SOLR_INCLUDE_FILE_OPTION);
         File includeFile = new File(solrIncludeFilename);
         if (!includeFile.exists() || !includeFile.canWrite()) {
           CLIO.out(
@@ -378,7 +394,7 @@ private int handleBasicAuth(CommandLine cli) throws Exception {
           printAuthEnablingInstructions(username, password);
           System.exit(0);
         }
-        String authConfDir = cli.getOptionValue("auth-conf-dir");
+        String authConfDir = cli.getOptionValue(AUTH_CONF_DIR_OPTION);
         File basicAuthConfFile = new File(authConfDir + File.separator + "basicAuth.conf");
 
         if (!basicAuthConfFile.getParentFile().canWrite()) {
@@ -394,7 +410,7 @@ private int handleBasicAuth(CommandLine cli) throws Exception {
 
         // update the solr.in.sh file to contain the necessary authentication lines
         updateIncludeFileEnableAuth(
-            includeFile.toPath(), basicAuthConfFile.getAbsolutePath(), null, cli);
+            includeFile.toPath(), basicAuthConfFile.getAbsolutePath(), null);
         final String successMessage =
             String.format(
                 Locale.ROOT,
@@ -402,12 +418,11 @@ private int handleBasicAuth(CommandLine cli) throws Exception {
                 username,
                 password);
         echo(successMessage);
-        return 0;
-
+        return;
       case "disable":
         clearSecurityJson(cli, updateIncludeFileOnly);
 
-        solrIncludeFilename = cli.getOptionValue("solr-include-file");
+        solrIncludeFilename = cli.getOptionValue(SOLR_INCLUDE_FILE_OPTION);
         includeFile = new File(solrIncludeFilename);
         if (!includeFile.exists() || !includeFile.canWrite()) {
           CLIO.out(
@@ -418,18 +433,15 @@ private int handleBasicAuth(CommandLine cli) throws Exception {
         }
 
         // update the solr.in.sh file to comment out the necessary authentication lines
-        updateIncludeFileDisableAuth(includeFile.toPath(), cli);
-        return 0;
-
+        updateIncludeFileDisableAuth(includeFile.toPath());
+        return;
       default:
         CLIO.out("Valid auth commands are: enable, disable.");
         SolrCLI.exit(1);
     }
 
     CLIO.out("Options not understood.");
-    new HelpFormatter()
-        .printHelp("bin/solr auth <enable|disable> [OPTIONS]", SolrCLI.getToolOptions(this));
-    return 1;
+    SolrCLI.exit(1);
   }
 
   private void checkSecurityJsonExists(SolrZkClient zkClient)
@@ -517,8 +529,7 @@ private void printAuthEnablingInstructions(String kerberosConfig) {
    *     null.
    */
   private void updateIncludeFileEnableAuth(
-      Path includeFile, String basicAuthConfFile, String kerberosConfig, CommandLine cli)
-      throws IOException {
+      Path includeFile, String basicAuthConfFile, String kerberosConfig) throws IOException {
     assert !(basicAuthConfFile != null
         && kerberosConfig != null); // only one of the two needs to be populated
     List<String> includeFileLines = Files.readAllLines(includeFile, StandardCharsets.UTF_8);
@@ -552,7 +563,7 @@ private void updateIncludeFileEnableAuth(
         includeFileLines.add("REM The following lines added by solr.cmd for enabling BasicAuth");
         includeFileLines.add("set SOLR_AUTH_TYPE=kerberos");
         includeFileLines.add(
-            "set SOLR_AUTHENTICATION_OPTS=\"-Dsolr.httpclient.config=" + basicAuthConfFile + "\"");
+            "set SOLR_AUTHENTICATION_OPTS=\"-Dsolr.httpclient.config=basicAuthConfFile\"");
       } else {
         includeFileLines.add("# The following lines added by ./solr for enabling BasicAuth");
         includeFileLines.add("SOLR_AUTH_TYPE=\"kerberos\"");
@@ -569,7 +580,7 @@ private void updateIncludeFileEnableAuth(
     echoIfVerbose("Updated Solr include file: " + includeFile.toAbsolutePath());
   }
 
-  private void updateIncludeFileDisableAuth(Path includeFile, CommandLine cli) throws IOException {
+  private void updateIncludeFileDisableAuth(Path includeFile) throws IOException {
     List<String> includeFileLines = Files.readAllLines(includeFile, StandardCharsets.UTF_8);
     boolean hasChanged = false;
     for (int i = 0; i < includeFileLines.size(); i++) {
@@ -594,12 +605,10 @@ private void updateIncludeFileDisableAuth(Path includeFile, CommandLine cli) thr
 
   @Override
   public void runImpl(CommandLine cli) throws Exception {
-    SolrCLI.raiseLogLevelUnlessVerbose(cli);
-
-    ensureArgumentIsValidBooleanIfPresent(cli, "block-unknown");
-    ensureArgumentIsValidBooleanIfPresent(cli, "update-include-file-only");
+    ensureArgumentIsValidBooleanIfPresent(cli, BLOCK_UNKNOWN_OPTION);
+    ensureArgumentIsValidBooleanIfPresent(cli, UPDATE_INCLUDE_FILE_OPTION);
 
-    String type = cli.getOptionValue("type", "basicAuth");
+    String type = cli.getOptionValue(TYPE_OPTION, "basicAuth");
     switch (type) {
       case "basicAuth":
         handleBasicAuth(cli);
diff --git a/solr/core/src/java/org/apache/solr/cli/ClusterTool.java b/solr/core/src/java/org/apache/solr/cli/ClusterTool.java
index 8b6bd17f56b..1c71b7b8fee 100644
--- a/solr/core/src/java/org/apache/solr/cli/ClusterTool.java
+++ b/solr/core/src/java/org/apache/solr/cli/ClusterTool.java
@@ -19,10 +19,10 @@
 
 import java.io.IOException;
 import java.io.PrintStream;
-import java.util.List;
 import java.util.concurrent.TimeUnit;
 import org.apache.commons.cli.CommandLine;
 import org.apache.commons.cli.Option;
+import org.apache.commons.cli.Options;
 import org.apache.solr.client.solrj.impl.SolrZkClientTimeout;
 import org.apache.solr.cloud.ZkController;
 import org.apache.solr.common.cloud.ClusterProperties;
@@ -36,6 +36,23 @@
 public class ClusterTool extends ToolBase {
   // It is a shame this tool doesn't more closely mimic how the ConfigTool works.
 
+  private static final Option PROPERTY_OPTION =
+      Option.builder()
+          .longOpt("property")
+          .hasArg()
+          .argName("PROPERTY")
+          .required()
+          .desc("Name of the Cluster property to apply the action to, such as: 'urlScheme'.")
+          .build();
+
+  private static final Option VALUE_OPTION =
+      Option.builder()
+          .longOpt("value")
+          .hasArg()
+          .argName("VALUE")
+          .desc("Set the property to this value.")
+          .build();
+
   public ClusterTool() {
     this(CLIO.getOutStream());
   }
@@ -50,30 +67,18 @@ public String getName() {
   }
 
   @Override
-  public List<Option> getOptions() {
-    return List.of(
-        Option.builder()
-            .longOpt("property")
-            .hasArg()
-            .argName("PROPERTY")
-            .required(true)
-            .desc("Name of the Cluster property to apply the action to, such as: 'urlScheme'.")
-            .build(),
-        Option.builder()
-            .longOpt("value")
-            .hasArg()
-            .argName("VALUE")
-            .required(false)
-            .desc("Set the property to this value.")
-            .build(),
-        SolrCLI.OPTION_ZKHOST);
+  public Options getOptions() {
+    return super.getOptions()
+        .addOption(PROPERTY_OPTION)
+        .addOption(VALUE_OPTION)
+        .addOption(CommonCLIOptions.ZK_HOST_OPTION);
   }
 
   @Override
   public void runImpl(CommandLine cli) throws Exception {
 
-    String propertyName = cli.getOptionValue("property");
-    String propertyValue = cli.getOptionValue("value");
+    String propertyName = cli.getOptionValue(PROPERTY_OPTION);
+    String propertyValue = cli.getOptionValue(VALUE_OPTION);
     String zkHost = SolrCLI.getZkHost(cli);
 
     if (!ZkController.checkChrootPath(zkHost, true)) {
diff --git a/solr/core/src/java/org/apache/solr/cli/CommonCLIOptions.java b/solr/core/src/java/org/apache/solr/cli/CommonCLIOptions.java
new file mode 100644
index 00000000000..93f02c3e359
--- /dev/null
+++ b/solr/core/src/java/org/apache/solr/cli/CommonCLIOptions.java
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.cli;
+
+import org.apache.commons.cli.Option;
+
+public final class CommonCLIOptions {
+
+  private CommonCLIOptions() {}
+
+  public static final Option VERBOSE_OPTION =
+      Option.builder("v").longOpt("verbose").desc("Enable verbose command output.").build();
+
+  public static final Option HELP_OPTION =
+      Option.builder("h").longOpt("help").desc("Print this message.").build();
+
+  public static final Option ZK_HOST_OPTION =
+      Option.builder("z")
+          .longOpt("zk-host")
+          .hasArg()
+          .argName("HOST")
+          .desc(
+              "Zookeeper connection string; unnecessary if ZK_HOST is defined in solr.in.sh; otherwise, defaults to "
+                  + DefaultValues.ZK_HOST
+                  + '.')
+          .build();
+
+  public static final Option SOLR_URL_OPTION =
+      Option.builder("s")
+          .longOpt("solr-url")
+          .hasArg()
+          .argName("HOST")
+          .desc(
+              "Base Solr URL, which can be used to determine the zk-host if that's not known; defaults to: "
+                  + SolrCLI.getDefaultSolrUrl()
+                  + '.')
+          .build();
+
+  public static final Option RECURSIVE_OPTION =
+      Option.builder("r").longOpt("recursive").desc("Apply the command recursively.").build();
+
+  public static final Option CREDENTIALS_OPTION =
+      Option.builder("u")
+          .longOpt("credentials")
+          .hasArg()
+          .argName("credentials")
+          .desc(
+              "Credentials in the format username:password. Example: --credentials solr:SolrRocks")
+          .build();
+
+  public static final class DefaultValues {
+
+    private DefaultValues() {}
+
+    public static final String ZK_HOST = "localhost:9983";
+
+    public static final String DEFAULT_CONFIG_SET = "_default";
+  }
+}
diff --git a/solr/core/src/java/org/apache/solr/cli/ConfigSetDownloadTool.java b/solr/core/src/java/org/apache/solr/cli/ConfigSetDownloadTool.java
index b0e5aaa6d4a..b55b939117b 100644
--- a/solr/core/src/java/org/apache/solr/cli/ConfigSetDownloadTool.java
+++ b/solr/core/src/java/org/apache/solr/cli/ConfigSetDownloadTool.java
@@ -21,9 +21,9 @@
 import java.nio.file.Files;
 import java.nio.file.Path;
 import java.nio.file.Paths;
-import java.util.List;
 import org.apache.commons.cli.CommandLine;
 import org.apache.commons.cli.Option;
+import org.apache.commons.cli.Options;
 import org.apache.solr.common.cloud.SolrZkClient;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
@@ -32,6 +32,24 @@
 public class ConfigSetDownloadTool extends ToolBase {
   private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
 
+  private static final Option CONF_NAME_OPTION =
+      Option.builder("n")
+          .longOpt("conf-name")
+          .hasArg()
+          .argName("NAME")
+          .required()
+          .desc("Configset name in ZooKeeper.")
+          .build();
+
+  private static final Option CONF_DIR_OPTION =
+      Option.builder("d")
+          .longOpt("conf-dir")
+          .hasArg()
+          .argName("DIR")
+          .required()
+          .desc("Local directory with configs.")
+          .build();
+
   public ConfigSetDownloadTool() {
     this(CLIO.getOutStream());
   }
@@ -41,25 +59,12 @@ public ConfigSetDownloadTool(PrintStream stdout) {
   }
 
   @Override
-  public List<Option> getOptions() {
-    return List.of(
-        Option.builder("n")
-            .longOpt("conf-name")
-            .hasArg()
-            .argName("NAME")
-            .required(true)
-            .desc("Configset name in ZooKeeper.")
-            .build(),
-        Option.builder("d")
-            .longOpt("conf-dir")
-            .hasArg()
-            .argName("DIR")
-            .required(true)
-            .desc("Local directory with configs.")
-            .build(),
-        SolrCLI.OPTION_SOLRURL,
-        SolrCLI.OPTION_ZKHOST,
-        SolrCLI.OPTION_CREDENTIALS);
+  public Options getOptions() {
+    return super.getOptions()
+        .addOption(CONF_NAME_OPTION)
+        .addOption(CONF_DIR_OPTION)
+        .addOption(CommonCLIOptions.CREDENTIALS_OPTION)
+        .addOptionGroup(getConnectionOptions());
   }
 
   @Override
@@ -74,11 +79,10 @@ public String getUsage() {
 
   @Override
   public void runImpl(CommandLine cli) throws Exception {
-    SolrCLI.raiseLogLevelUnlessVerbose(cli);
     String zkHost = SolrCLI.getZkHost(cli);
 
-    String confName = cli.getOptionValue("conf-name");
-    String confDir = cli.getOptionValue("conf-dir");
+    String confName = cli.getOptionValue(CONF_NAME_OPTION);
+    String confDir = cli.getOptionValue(CONF_DIR_OPTION);
 
     echoIfVerbose("\nConnecting to ZooKeeper at " + zkHost + " ...");
     try (SolrZkClient zkClient = SolrCLI.getSolrZkClient(cli, zkHost)) {
diff --git a/solr/core/src/java/org/apache/solr/cli/ConfigSetUploadTool.java b/solr/core/src/java/org/apache/solr/cli/ConfigSetUploadTool.java
index 597b901c7d0..b959245af2e 100644
--- a/solr/core/src/java/org/apache/solr/cli/ConfigSetUploadTool.java
+++ b/solr/core/src/java/org/apache/solr/cli/ConfigSetUploadTool.java
@@ -20,9 +20,9 @@
 import java.lang.invoke.MethodHandles;
 import java.nio.file.Path;
 import java.nio.file.Paths;
-import java.util.List;
 import org.apache.commons.cli.CommandLine;
 import org.apache.commons.cli.Option;
+import org.apache.commons.cli.Options;
 import org.apache.solr.common.cloud.SolrZkClient;
 import org.apache.solr.common.cloud.ZkMaintenanceUtils;
 import org.apache.solr.core.ConfigSetService;
@@ -34,6 +34,24 @@
 public class ConfigSetUploadTool extends ToolBase {
   private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
 
+  private static final Option CONF_NAME_OPTION =
+      Option.builder("n")
+          .longOpt("conf-name")
+          .hasArg()
+          .argName("NAME")
+          .required()
+          .desc("Configset name in ZooKeeper.")
+          .build();
+
+  private static final Option CONF_DIR_OPTION =
+      Option.builder("d")
+          .longOpt("conf-dir")
+          .hasArg()
+          .argName("DIR")
+          .required()
+          .desc("Local directory with configs.")
+          .build();
+
   public ConfigSetUploadTool() {
     this(CLIO.getOutStream());
   }
@@ -43,25 +61,12 @@ public ConfigSetUploadTool(PrintStream stdout) {
   }
 
   @Override
-  public List<Option> getOptions() {
-    return List.of(
-        Option.builder("n")
-            .longOpt("conf-name")
-            .hasArg()
-            .argName("NAME")
-            .required(true)
-            .desc("Configset name in ZooKeeper.")
-            .build(),
-        Option.builder("d")
-            .longOpt("conf-dir")
-            .hasArg()
-            .argName("DIR")
-            .required(true)
-            .desc("Local directory with configs.")
-            .build(),
-        SolrCLI.OPTION_SOLRURL,
-        SolrCLI.OPTION_ZKHOST,
-        SolrCLI.OPTION_CREDENTIALS);
+  public Options getOptions() {
+    return super.getOptions()
+        .addOption(CONF_NAME_OPTION)
+        .addOption(CONF_DIR_OPTION)
+        .addOption(CommonCLIOptions.CREDENTIALS_OPTION)
+        .addOptionGroup(getConnectionOptions());
   }
 
   @Override
@@ -76,14 +81,13 @@ public String getUsage() {
 
   @Override
   public void runImpl(CommandLine cli) throws Exception {
-    SolrCLI.raiseLogLevelUnlessVerbose(cli);
     String zkHost = SolrCLI.getZkHost(cli);
 
     final String solrInstallDir = System.getProperty("solr.install.dir");
     Path solrInstallDirPath = Paths.get(solrInstallDir);
 
-    String confName = cli.getOptionValue("conf-name");
-    String confDir = cli.getOptionValue("conf-dir");
+    String confName = cli.getOptionValue(CONF_NAME_OPTION);
+    String confDir = cli.getOptionValue(CONF_DIR_OPTION);
 
     echoIfVerbose("\nConnecting to ZooKeeper at " + zkHost + " ...");
     try (SolrZkClient zkClient = SolrCLI.getSolrZkClient(cli, zkHost)) {
diff --git a/solr/core/src/java/org/apache/solr/cli/ConfigTool.java b/solr/core/src/java/org/apache/solr/cli/ConfigTool.java
index 5cc3f0909d6..fdd18f24d4b 100644
--- a/solr/core/src/java/org/apache/solr/cli/ConfigTool.java
+++ b/solr/core/src/java/org/apache/solr/cli/ConfigTool.java
@@ -19,11 +19,11 @@
 
 import java.io.PrintStream;
 import java.util.HashMap;
-import java.util.List;
 import java.util.Map;
 import org.apache.commons.cli.CommandLine;
 import org.apache.commons.cli.MissingArgumentException;
 import org.apache.commons.cli.Option;
+import org.apache.commons.cli.Options;
 import org.apache.solr.client.solrj.SolrClient;
 import org.apache.solr.common.util.NamedList;
 import org.noggit.CharArr;
@@ -36,6 +36,42 @@
  */
 public class ConfigTool extends ToolBase {
 
+  private static final Option COLLECTION_NAME_OPTION =
+      Option.builder("c")
+          .longOpt("name")
+          .hasArg()
+          .argName("NAME")
+          .required()
+          .desc("Name of the collection.")
+          .build();
+
+  private static final Option ACTION_OPTION =
+      Option.builder("a")
+          .longOpt("action")
+          .hasArg()
+          .argName("ACTION")
+          .desc(
+              "Config API action, one of: set-property, unset-property, set-user-property, unset-user-property; default is 'set-property'.")
+          .build();
+
+  private static final Option PROPERTY_OPTION =
+      Option.builder()
+          .longOpt("property")
+          .hasArg()
+          .argName("PROP")
+          .required()
+          .desc(
+              "Name of the Config API property to apply the action to, such as: 'updateHandler.autoSoftCommit.maxTime'.")
+          .build();
+
+  private static final Option VALUE_OPTION =
+      Option.builder("v")
+          .longOpt("value")
+          .hasArg()
+          .argName("VALUE")
+          .desc("Set the property to this value; accepts JSON objects and strings.")
+          .build();
+
   public ConfigTool() {
     this(CLIO.getOutStream());
   }
@@ -50,50 +86,23 @@ public String getName() {
   }
 
   @Override
-  public List<Option> getOptions() {
-    return List.of(
-        Option.builder("c")
-            .longOpt("name")
-            .argName("NAME")
-            .hasArg()
-            .required(true)
-            .desc("Name of the collection.")
-            .build(),
-        Option.builder("a")
-            .longOpt("action")
-            .argName("ACTION")
-            .hasArg()
-            .required(false)
-            .desc(
-                "Config API action, one of: set-property, unset-property, set-user-property, unset-user-property; default is 'set-property'.")
-            .build(),
-        Option.builder()
-            .longOpt("property")
-            .argName("PROP")
-            .hasArg()
-            .required(true)
-            .desc(
-                "Name of the Config API property to apply the action to, such as: 'updateHandler.autoSoftCommit.maxTime'.")
-            .build(),
-        Option.builder()
-            .longOpt("value")
-            .argName("VALUE")
-            .hasArg()
-            .required(false)
-            .desc("Set the property to this value; accepts JSON objects and strings.")
-            .build(),
-        SolrCLI.OPTION_SOLRURL,
-        SolrCLI.OPTION_ZKHOST,
-        SolrCLI.OPTION_CREDENTIALS);
+  public Options getOptions() {
+    return super.getOptions()
+        .addOption(COLLECTION_NAME_OPTION)
+        .addOption(ACTION_OPTION)
+        .addOption(PROPERTY_OPTION)
+        .addOption(VALUE_OPTION)
+        .addOption(CommonCLIOptions.CREDENTIALS_OPTION)
+        .addOptionGroup(getConnectionOptions());
   }
 
   @Override
   public void runImpl(CommandLine cli) throws Exception {
     String solrUrl = SolrCLI.normalizeSolrUrl(cli);
-    String action = cli.getOptionValue("action", "set-property");
-    String collection = cli.getOptionValue("name");
-    String property = cli.getOptionValue("property");
-    String value = cli.getOptionValue("value");
+    String action = cli.getOptionValue(ACTION_OPTION, "set-property");
+    String collection = cli.getOptionValue(COLLECTION_NAME_OPTION);
+    String property = cli.getOptionValue(PROPERTY_OPTION);
+    String value = cli.getOptionValue(VALUE_OPTION);
 
     // value is required unless the property is one of the unset- type.
     if (!action.contains("unset-") && value == null) {
@@ -118,8 +127,7 @@ public void runImpl(CommandLine cli) throws Exception {
     echoIfVerbose(jsonBody);
 
     try (SolrClient solrClient =
-        SolrCLI.getSolrClient(
-            solrUrl, cli.getOptionValue(SolrCLI.OPTION_CREDENTIALS.getLongOpt()))) {
+        SolrCLI.getSolrClient(solrUrl, cli.getOptionValue(CommonCLIOptions.CREDENTIALS_OPTION))) {
       NamedList<Object> result = SolrCLI.postJsonToSolr(solrClient, updatePath, jsonBody);
       Integer statusCode = (Integer) result.findRecursive("responseHeader", "status");
       if (statusCode == 0) {
diff --git a/solr/core/src/java/org/apache/solr/cli/CreateTool.java b/solr/core/src/java/org/apache/solr/cli/CreateTool.java
index 6c14ed6e21e..5fc57501d93 100644
--- a/solr/core/src/java/org/apache/solr/cli/CreateTool.java
+++ b/solr/core/src/java/org/apache/solr/cli/CreateTool.java
@@ -21,15 +21,16 @@
 import java.nio.file.Files;
 import java.nio.file.Path;
 import java.nio.file.Paths;
-import java.util.List;
 import java.util.Locale;
 import java.util.Map;
 import java.util.Set;
 import java.util.concurrent.TimeUnit;
 import org.apache.commons.cli.CommandLine;
 import org.apache.commons.cli.Option;
+import org.apache.commons.cli.Options;
 import org.apache.commons.io.FileUtils;
 import org.apache.commons.io.file.PathUtils;
+import org.apache.solr.cli.CommonCLIOptions.DefaultValues;
 import org.apache.solr.client.solrj.SolrClient;
 import org.apache.solr.client.solrj.SolrRequest;
 import org.apache.solr.client.solrj.SolrServerException;
@@ -52,6 +53,53 @@
 /** Supports create command in the bin/solr script. */
 public class CreateTool extends ToolBase {
 
+  private static final Option COLLECTION_NAME_OPTION =
+      Option.builder("c")
+          .longOpt("name")
+          .hasArg()
+          .argName("NAME")
+          .required()
+          .desc("Name of collection or core to create.")
+          .build();
+
+  private static final Option SHARDS_OPTION =
+      Option.builder("sh")
+          .longOpt("shards")
+          .hasArg()
+          .argName("#")
+          .type(Integer.class)
+          .desc("Number of shards; default is 1.")
+          .build();
+
+  private static final Option REPLICATION_FACTOR_OPTION =
+      Option.builder("rf")
+          .longOpt("replication-factor")
+          .hasArg()
+          .argName("#")
+          .type(Integer.class)
+          .desc(
+              "Number of copies of each document across the collection (replicas per shard); default is 1.")
+          .build();
+
+  private static final Option CONF_DIR_OPTION =
+      Option.builder("d")
+          .longOpt("conf-dir")
+          .hasArg()
+          .argName("DIR")
+          .desc(
+              "Configuration directory to copy when creating the new collection; default is "
+                  + DefaultValues.DEFAULT_CONFIG_SET
+                  + '.')
+          .build();
+
+  private static final Option CONF_NAME_OPTION =
+      Option.builder("n")
+          .longOpt("conf-name")
+          .hasArg()
+          .argName("NAME")
+          .desc("Configuration name; default is the collection name.")
+          .build();
+
   public CreateTool() {
     this(CLIO.getOutStream());
   }
@@ -74,53 +122,22 @@ public String getHeader() {
   }
 
   @Override
-  public List<Option> getOptions() {
-    return List.of(
-        Option.builder("c")
-            .longOpt("name")
-            .hasArg()
-            .argName("NAME")
-            .required(true)
-            .desc("Name of collection or core to create.")
-            .build(),
-        Option.builder("sh")
-            .longOpt("shards")
-            .hasArg()
-            .argName("#")
-            .desc("Number of shards; default is 1.")
-            .build(),
-        Option.builder("rf")
-            .longOpt("replication-factor")
-            .hasArg()
-            .argName("#")
-            .desc(
-                "Number of copies of each document across the collection (replicas per shard); default is 1.")
-            .build(),
-        Option.builder("d")
-            .longOpt("conf-dir")
-            .argName("DIR")
-            .hasArg()
-            .desc(
-                "Configuration directory to copy when creating the new collection; default is "
-                    + SolrCLI.DEFAULT_CONFIG_SET
-                    + '.')
-            .build(),
-        Option.builder("n")
-            .longOpt("conf-name")
-            .argName("NAME")
-            .hasArg()
-            .required(false)
-            .desc("Configuration name; default is the collection name.")
-            .build(),
-        SolrCLI.OPTION_SOLRURL,
-        SolrCLI.OPTION_ZKHOST,
-        SolrCLI.OPTION_CREDENTIALS);
+  public Options getOptions() {
+    Options opts =
+        super.getOptions()
+            .addOption(COLLECTION_NAME_OPTION)
+            .addOption(SHARDS_OPTION)
+            .addOption(REPLICATION_FACTOR_OPTION)
+            .addOption(CONF_DIR_OPTION)
+            .addOption(CONF_NAME_OPTION)
+            .addOption(CommonCLIOptions.CREDENTIALS_OPTION)
+            .addOptionGroup(getConnectionOptions());
+
+    return opts;
   }
 
   @Override
   public void runImpl(CommandLine cli) throws Exception {
-    SolrCLI.raiseLogLevelUnlessVerbose(cli);
-
     try (var solrClient = SolrCLI.getSolrClient(cli)) {
       if (SolrCLI.isCloudMode(solrClient)) {
         createCollection(cli);
@@ -131,11 +148,13 @@ public void runImpl(CommandLine cli) throws Exception {
   }
 
   protected void createCore(CommandLine cli, SolrClient solrClient) throws Exception {
-    String coreName = cli.getOptionValue("name");
-    String solrUrl = cli.getOptionValue("solr-url", SolrCLI.getDefaultSolrUrl());
+    String coreName = cli.getOptionValue(COLLECTION_NAME_OPTION);
+    String solrUrl =
+        cli.getOptionValue(CommonCLIOptions.SOLR_URL_OPTION, SolrCLI.getDefaultSolrUrl());
 
     final String solrInstallDir = System.getProperty("solr.install.dir");
-    final String confDirName = cli.getOptionValue("conf-dir", SolrCLI.DEFAULT_CONFIG_SET);
+    final String confDirName =
+        cli.getOptionValue(CONF_DIR_OPTION, DefaultValues.DEFAULT_CONFIG_SET);
 
     // we allow them to pass a directory instead of a configset name
     Path configsetDir = Paths.get(confDirName);
@@ -157,7 +176,7 @@ protected void createCore(CommandLine cli, SolrClient solrClient) throws Excepti
     coreRootDirectory = (String) systemInfo.get("core_root");
 
     if (SolrCLI.safeCheckCoreExists(
-        solrUrl, coreName, cli.getOptionValue(SolrCLI.OPTION_CREDENTIALS.getLongOpt()))) {
+        solrUrl, coreName, cli.getOptionValue(CommonCLIOptions.CREDENTIALS_OPTION))) {
       throw new IllegalArgumentException(
           "\nCore '"
               + coreName
@@ -204,7 +223,7 @@ protected void createCollection(CommandLine cli) throws Exception {
             .withConnectionTimeout(15, TimeUnit.SECONDS)
             .withKeyStoreReloadInterval(-1, TimeUnit.SECONDS)
             .withOptionalBasicAuthCredentials(
-                cli.getOptionValue(SolrCLI.OPTION_CREDENTIALS.getLongOpt()));
+                cli.getOptionValue(CommonCLIOptions.CREDENTIALS_OPTION));
     String zkHost = SolrCLI.getZkHost(cli);
     echoIfVerbose("Connecting to ZooKeeper at " + zkHost);
     try (CloudSolrClient cloudSolrClient = SolrCLI.getCloudHttp2SolrClient(zkHost, builder)) {
@@ -216,10 +235,10 @@ protected void createCollection(CommandLine cli) throws Exception {
   protected void createCollection(CloudSolrClient cloudSolrClient, CommandLine cli)
       throws Exception {
 
-    String collectionName = cli.getOptionValue("name");
+    String collectionName = cli.getOptionValue(COLLECTION_NAME_OPTION);
     final String solrInstallDir = System.getProperty("solr.install.dir");
-    String confName = cli.getOptionValue("conf-name");
-    String confDir = cli.getOptionValue("conf-dir", SolrCLI.DEFAULT_CONFIG_SET);
+    String confName = cli.getOptionValue(CONF_NAME_OPTION);
+    String confDir = cli.getOptionValue(CONF_DIR_OPTION, DefaultValues.DEFAULT_CONFIG_SET);
     Path solrInstallDirPath = Paths.get(solrInstallDir);
     Path confDirPath = Paths.get(confDir);
     ensureConfDirExists(solrInstallDirPath, confDirPath);
@@ -231,19 +250,15 @@ protected void createCollection(CloudSolrClient cloudSolrClient, CommandLine cli
           "No live nodes found! Cannot create a collection until "
               + "there is at least 1 live node in the cluster.");
 
-    String solrUrl = cli.getOptionValue("solr-url");
+    String solrUrl = cli.getOptionValue(CommonCLIOptions.SOLR_URL_OPTION);
     if (solrUrl == null) {
       String firstLiveNode = liveNodes.iterator().next();
       solrUrl = ZkStateReader.from(cloudSolrClient).getBaseUrlForNodeName(firstLiveNode);
     }
 
     // build a URL to create the collection
-    int numShards = Integer.parseInt(cli.getOptionValue("shards", String.valueOf(1)));
-    int replicationFactor = 1;
-
-    if (cli.hasOption("replication-factor")) {
-      replicationFactor = Integer.parseInt(cli.getOptionValue("replication-factor"));
-    }
+    int numShards = cli.getParsedOptionValue(SHARDS_OPTION, 1);
+    int replicationFactor = cli.getParsedOptionValue(REPLICATION_FACTOR_OPTION, 1);
 
     boolean configExistsInZk =
         confName != null
@@ -280,7 +295,7 @@ protected void createCollection(CloudSolrClient cloudSolrClient, CommandLine cli
 
     // since creating a collection is a heavy-weight operation, check for existence first
     if (SolrCLI.safeCheckCollectionExists(
-        solrUrl, collectionName, cli.getOptionValue(SolrCLI.OPTION_CREDENTIALS.getLongOpt()))) {
+        solrUrl, collectionName, cli.getOptionValue(CommonCLIOptions.CREDENTIALS_OPTION))) {
       throw new IllegalStateException(
           "\nCollection '"
               + collectionName
@@ -339,13 +354,15 @@ private void ensureConfDirExists(Path solrInstallDir, Path confDirName) {
   }
 
   private void printDefaultConfigsetWarningIfNecessary(CommandLine cli) {
-    final String confDirectoryName = cli.getOptionValue("conf-dir", SolrCLI.DEFAULT_CONFIG_SET);
-    final String confName = cli.getOptionValue("conf-name", "");
+    final String confDirectoryName =
+        cli.getOptionValue(CONF_DIR_OPTION, DefaultValues.DEFAULT_CONFIG_SET);
+    final String confName = cli.getOptionValue(CONF_NAME_OPTION, "");
 
     if (confDirectoryName.equals("_default")
         && (confName.equals("") || confName.equals("_default"))) {
-      final String collectionName = cli.getOptionValue("name");
-      final String solrUrl = cli.getOptionValue("solr-url", SolrCLI.getDefaultSolrUrl());
+      final String collectionName = cli.getOptionValue(COLLECTION_NAME_OPTION);
+      final String solrUrl =
+          cli.getOptionValue(CommonCLIOptions.SOLR_URL_OPTION, SolrCLI.getDefaultSolrUrl());
       final String curlCommand =
           String.format(
               Locale.ROOT,
diff --git a/solr/core/src/java/org/apache/solr/cli/DeleteTool.java b/solr/core/src/java/org/apache/solr/cli/DeleteTool.java
index bd6d5f97cc2..61e02161416 100644
--- a/solr/core/src/java/org/apache/solr/cli/DeleteTool.java
+++ b/solr/core/src/java/org/apache/solr/cli/DeleteTool.java
@@ -16,18 +16,16 @@
  */
 package org.apache.solr.cli;
 
-import static org.apache.solr.common.params.CommonParams.NAME;
-
 import java.io.PrintStream;
 import java.lang.invoke.MethodHandles;
 import java.util.Collection;
-import java.util.List;
 import java.util.Locale;
 import java.util.Optional;
 import java.util.Set;
 import java.util.concurrent.TimeUnit;
 import org.apache.commons.cli.CommandLine;
 import org.apache.commons.cli.Option;
+import org.apache.commons.cli.Options;
 import org.apache.solr.client.solrj.SolrClient;
 import org.apache.solr.client.solrj.SolrServerException;
 import org.apache.solr.client.solrj.impl.CloudSolrClient;
@@ -47,6 +45,32 @@
 public class DeleteTool extends ToolBase {
   private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
 
+  private static final Option COLLECTION_NAME_OPTION =
+      Option.builder("c")
+          .longOpt("name")
+          .hasArg()
+          .argName("NAME")
+          .required()
+          .desc("Name of the core / collection to delete.")
+          .build();
+
+  private static final Option DELETE_CONFIG_OPTION =
+      Option.builder()
+          .longOpt("delete-config")
+          .hasArg()
+          .argName("true|false")
+          .type(Boolean.class)
+          .desc(
+              "Flag to indicate if the underlying configuration directory for a collection should also be deleted; default is true.")
+          .build();
+
+  private static final Option FORCE_OPTION =
+      Option.builder("f")
+          .longOpt("force")
+          .desc(
+              "Skip safety checks when deleting the configuration directory used by a collection.")
+          .build();
+
   public DeleteTool() {
     this(CLIO.getOutStream());
   }
@@ -71,38 +95,17 @@ public String getHeader() {
   }
 
   @Override
-  public List<Option> getOptions() {
-    return List.of(
-        Option.builder("c")
-            .longOpt("name")
-            .argName("NAME")
-            .hasArg()
-            .required(true)
-            .desc("Name of the core / collection to delete.")
-            .build(),
-        Option.builder()
-            .longOpt("delete-config")
-            .hasArg()
-            .argName("true|false")
-            .required(false)
-            .desc(
-                "Flag to indicate if the underlying configuration directory for a collection should also be deleted; default is true.")
-            .build(),
-        Option.builder("f")
-            .longOpt("force")
-            .required(false)
-            .desc(
-                "Skip safety checks when deleting the configuration directory used by a collection.")
-            .build(),
-        SolrCLI.OPTION_SOLRURL,
-        SolrCLI.OPTION_ZKHOST,
-        SolrCLI.OPTION_CREDENTIALS);
+  public Options getOptions() {
+    return super.getOptions()
+        .addOption(COLLECTION_NAME_OPTION)
+        .addOption(DELETE_CONFIG_OPTION)
+        .addOption(FORCE_OPTION)
+        .addOption(CommonCLIOptions.CREDENTIALS_OPTION)
+        .addOptionGroup(getConnectionOptions());
   }
 
   @Override
   public void runImpl(CommandLine cli) throws Exception {
-    SolrCLI.raiseLogLevelUnlessVerbose(cli);
-
     try (var solrClient = SolrCLI.getSolrClient(cli)) {
       if (SolrCLI.isCloudMode(solrClient)) {
         deleteCollection(cli);
@@ -118,7 +121,8 @@ protected void deleteCollection(CommandLine cli) throws Exception {
             .withIdleTimeout(30, TimeUnit.SECONDS)
             .withConnectionTimeout(15, TimeUnit.SECONDS)
             .withKeyStoreReloadInterval(-1, TimeUnit.SECONDS)
-            .withOptionalBasicAuthCredentials(cli.getOptionValue(("credentials")));
+            .withOptionalBasicAuthCredentials(
+                cli.getOptionValue(CommonCLIOptions.CREDENTIALS_OPTION));
 
     String zkHost = SolrCLI.getZkHost(cli);
     try (CloudSolrClient cloudSolrClient = SolrCLI.getCloudHttp2SolrClient(zkHost, builder)) {
@@ -137,20 +141,17 @@ protected void deleteCollection(CloudSolrClient cloudSolrClient, CommandLine cli
               + "there is at least 1 live node in the cluster.");
 
     ZkStateReader zkStateReader = ZkStateReader.from(cloudSolrClient);
-    String collectionName = cli.getOptionValue(NAME);
+    String collectionName = cli.getOptionValue(COLLECTION_NAME_OPTION);
     if (!zkStateReader.getClusterState().hasCollection(collectionName)) {
       throw new IllegalArgumentException("Collection " + collectionName + " not found!");
     }
 
     String configName =
         zkStateReader.getClusterState().getCollection(collectionName).getConfigName();
-    boolean deleteConfig = true;
-    if (cli.hasOption("delete-config")) {
-      deleteConfig = "true".equals(cli.getOptionValue("delete-config"));
-    }
+    boolean deleteConfig = Boolean.parseBoolean(cli.getOptionValue(DELETE_CONFIG_OPTION, "true"));
 
     if (deleteConfig && configName != null) {
-      if (cli.hasOption("force")) {
+      if (cli.hasOption(FORCE_OPTION)) {
         log.warn(
             "Skipping safety checks, configuration directory {} will be deleted with impunity.",
             configName);
@@ -224,7 +225,7 @@ protected void deleteCollection(CloudSolrClient cloudSolrClient, CommandLine cli
   }
 
   protected void deleteCore(CommandLine cli, SolrClient solrClient) throws Exception {
-    String coreName = cli.getOptionValue(NAME);
+    String coreName = cli.getOptionValue(COLLECTION_NAME_OPTION);
 
     echo("\nDeleting core '" + coreName + "' using CoreAdminRequest\n");
 
diff --git a/solr/core/src/java/org/apache/solr/cli/ExportTool.java b/solr/core/src/java/org/apache/solr/cli/ExportTool.java
index 699df52d330..5018855e597 100644
--- a/solr/core/src/java/org/apache/solr/cli/ExportTool.java
+++ b/solr/core/src/java/org/apache/solr/cli/ExportTool.java
@@ -55,6 +55,7 @@
 import java.util.zip.GZIPOutputStream;
 import org.apache.commons.cli.CommandLine;
 import org.apache.commons.cli.Option;
+import org.apache.commons.cli.Options;
 import org.apache.lucene.util.SuppressForbidden;
 import org.apache.solr.client.solrj.SolrClient;
 import org.apache.solr.client.solrj.SolrQuery;
@@ -88,57 +89,76 @@
 
 /** Supports export command in the bin/solr script. */
 public class ExportTool extends ToolBase {
+
+  private static final Option COLLECTION_NAME_OPTION =
+      Option.builder("c")
+          .longOpt("name")
+          .hasArg()
+          .argName("NAME")
+          .desc("Name of the collection.")
+          .build();
+
+  private static final Option OUTPUT_OPTION =
+      Option.builder()
+          .longOpt("output")
+          .hasArg()
+          .argName("PATH")
+          .desc(
+              "Path to output the exported data, and optionally the file name, defaults to 'collection-name'.")
+          .build();
+
+  private static final Option FORMAT_OPTION =
+      Option.builder()
+          .longOpt("format")
+          .hasArg()
+          .argName("FORMAT")
+          .desc("Output format for exported docs (json, jsonl or javabin), defaulting to json.")
+          .build();
+
+  private static final Option COMPRESS_OPTION =
+      Option.builder().longOpt("compress").desc("Compress the output. Defaults to false.").build();
+
+  private static final Option LIMIT_OPTION =
+      Option.builder()
+          .longOpt("limit")
+          .hasArg()
+          .argName("#")
+          .desc("Maximum number of docs to download. Default is 100, use -1 for all docs.")
+          .build();
+
+  private static final Option QUERY_OPTION =
+      Option.builder()
+          .longOpt("query")
+          .hasArg()
+          .argName("QUERY")
+          .desc("A custom query, default is '*:*'.")
+          .build();
+
+  private static final Option FIELDS_OPTION =
+      Option.builder()
+          .longOpt("fields")
+          .hasArg()
+          .argName("FIELDA,FIELDB")
+          .desc("Comma separated list of fields to export. By default all fields are fetched.")
+          .build();
+
   @Override
   public String getName() {
     return "export";
   }
 
   @Override
-  public List<Option> getOptions() {
-    return List.of(
-        Option.builder("c")
-            .longOpt("name")
-            .hasArg()
-            .argName("NAME")
-            .desc("Name of the collection.")
-            .build(),
-        Option.builder()
-            .longOpt("output")
-            .hasArg()
-            .argName("PATH")
-            .desc(
-                "Path to output the exported data, and optionally the file name, defaults to 'collection-name'.")
-            .build(),
-        Option.builder()
-            .longOpt("format")
-            .hasArg()
-            .argName("FORMAT")
-            .desc("Output format for exported docs (json, jsonl or javabin), defaulting to json.")
-            .build(),
-        Option.builder()
-            .longOpt("compress")
-            .desc("Compress the output. Defaults to false.")
-            .build(),
-        Option.builder()
-            .longOpt("limit")
-            .hasArg()
-            .argName("#")
-            .desc("Maximum number of docs to download. Default is 100, use -1 for all docs.")
-            .build(),
-        Option.builder()
-            .longOpt("query")
-            .hasArg()
-            .argName("QUERY")
-            .desc("A custom query, default is '*:*'.")
-            .build(),
-        Option.builder()
-            .longOpt("fields")
-            .hasArg()
-            .argName("FIELDA,FIELDB")
-            .desc("Comma separated list of fields to export. By default all fields are fetched.")
-            .build(),
-        SolrCLI.OPTION_SOLRURL,
-        SolrCLI.OPTION_CREDENTIALS);
+  public Options getOptions() {
+    return super.getOptions()
+        .addOption(COLLECTION_NAME_OPTION)
+        .addOption(OUTPUT_OPTION)
+        .addOption(FORMAT_OPTION)
+        .addOption(COMPRESS_OPTION)
+        .addOption(LIMIT_OPTION)
+        .addOption(QUERY_OPTION)
+        .addOption(FIELDS_OPTION)
+        .addOption(CommonCLIOptions.SOLR_URL_OPTION)
+        .addOption(CommonCLIOptions.CREDENTIALS_OPTION);
   }
 
   public abstract static class Info {
@@ -259,25 +279,27 @@ public void streamDocListInfo(long numFound, long start, Float maxScore) {}
   @Override
   public void runImpl(CommandLine cli) throws Exception {
     String url = null;
-    if (cli.hasOption("solr-url")) {
-      if (!cli.hasOption("name")) {
+    if (cli.hasOption(CommonCLIOptions.SOLR_URL_OPTION)) {
+      if (!cli.hasOption(COLLECTION_NAME_OPTION)) {
         throw new IllegalArgumentException(
             "Must specify -c / --name parameter with --solr-url to post documents.");
       }
-      url = SolrCLI.normalizeSolrUrl(cli) + "/solr/" + cli.getOptionValue("name");
+      url = SolrCLI.normalizeSolrUrl(cli) + "/solr/" + cli.getOptionValue(COLLECTION_NAME_OPTION);
 
     } else {
       // think about support --zk-host someday.
       throw new IllegalArgumentException("Must specify --solr-url.");
     }
-    String credentials = cli.getOptionValue(SolrCLI.OPTION_CREDENTIALS.getLongOpt());
+    String credentials = cli.getOptionValue(CommonCLIOptions.CREDENTIALS_OPTION);
     Info info = new MultiThreadedRunner(url, credentials);
-    info.query = cli.getOptionValue("query", "*:*");
+    info.query = cli.getOptionValue(QUERY_OPTION, "*:*");
 
     info.setOutFormat(
-        cli.getOptionValue("output"), cli.getOptionValue("format"), cli.hasOption("compress"));
-    info.fields = cli.getOptionValue("fields");
-    info.setLimit(cli.getOptionValue("limit", "100"));
+        cli.getOptionValue(OUTPUT_OPTION),
+        cli.getOptionValue(FORMAT_OPTION),
+        cli.hasOption(COMPRESS_OPTION));
+    info.fields = cli.getOptionValue(FIELDS_OPTION);
+    info.setLimit(cli.getOptionValue(LIMIT_OPTION, "100"));
     info.output = super.stdout;
     info.exportDocs();
   }
diff --git a/solr/core/src/java/org/apache/solr/cli/HealthcheckTool.java b/solr/core/src/java/org/apache/solr/cli/HealthcheckTool.java
index dcc3aa91d11..6ec08f77d6c 100644
--- a/solr/core/src/java/org/apache/solr/cli/HealthcheckTool.java
+++ b/solr/core/src/java/org/apache/solr/cli/HealthcheckTool.java
@@ -30,6 +30,7 @@
 import java.util.Set;
 import org.apache.commons.cli.CommandLine;
 import org.apache.commons.cli.Option;
+import org.apache.commons.cli.Options;
 import org.apache.solr.client.solrj.SolrQuery;
 import org.apache.solr.client.solrj.SolrRequest;
 import org.apache.solr.client.solrj.impl.CloudHttp2SolrClient;
@@ -53,19 +54,21 @@
 public class HealthcheckTool extends ToolBase {
   private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
 
+  private static final Option COLLECTION_NAME_OPTION =
+      Option.builder("c")
+          .longOpt("name")
+          .hasArg()
+          .argName("COLLECTION")
+          .required()
+          .desc("Name of the collection to check.")
+          .build();
+
   @Override
-  public List<Option> getOptions() {
-    return List.of(
-        Option.builder("c")
-            .longOpt("name")
-            .hasArg()
-            .argName("COLLECTION")
-            .required(true)
-            .desc("Name of the collection to check.")
-            .build(),
-        SolrCLI.OPTION_SOLRURL,
-        SolrCLI.OPTION_ZKHOST,
-        SolrCLI.OPTION_CREDENTIALS);
+  public Options getOptions() {
+    return super.getOptions()
+        .addOption(COLLECTION_NAME_OPTION)
+        .addOption(CommonCLIOptions.CREDENTIALS_OPTION)
+        .addOptionGroup(getConnectionOptions());
   }
 
   enum ShardState {
@@ -86,7 +89,6 @@ public HealthcheckTool(PrintStream stdout) {
 
   @Override
   public void runImpl(CommandLine cli) throws Exception {
-    SolrCLI.raiseLogLevelUnlessVerbose(cli);
     String zkHost = SolrCLI.getZkHost(cli);
     if (zkHost == null) {
       CLIO.err("Healthcheck tool only works in Solr Cloud mode.");
@@ -105,8 +107,7 @@ public String getName() {
   }
 
   protected void runCloudTool(CloudSolrClient cloudSolrClient, CommandLine cli) throws Exception {
-    SolrCLI.raiseLogLevelUnlessVerbose(cli);
-    String collection = cli.getOptionValue("name");
+    String collection = cli.getOptionValue(COLLECTION_NAME_OPTION);
 
     log.debug("Running healthcheck for {}", collection);
 
@@ -169,13 +170,13 @@ protected void runCloudTool(CloudSolrClient cloudSolrClient, CommandLine cli) th
           q.set(DISTRIB, "false");
           try (var solrClientForCollection =
               SolrCLI.getSolrClient(
-                  coreUrl, cli.getOptionValue(SolrCLI.OPTION_CREDENTIALS.getLongOpt()))) {
+                  coreUrl, cli.getOptionValue(CommonCLIOptions.CREDENTIALS_OPTION))) {
             qr = solrClientForCollection.query(q);
             numDocs = qr.getResults().getNumFound();
             try (var solrClient =
                 SolrCLI.getSolrClient(
                     replicaCoreProps.getBaseUrl(),
-                    cli.getOptionValue(SolrCLI.OPTION_CREDENTIALS.getLongOpt()))) {
+                    cli.getOptionValue(CommonCLIOptions.CREDENTIALS_OPTION))) {
               NamedList<Object> systemInfo =
                   solrClient.request(
                       new GenericSolrRequest(
diff --git a/solr/core/src/java/org/apache/solr/cli/LinkConfigTool.java b/solr/core/src/java/org/apache/solr/cli/LinkConfigTool.java
index da553eb735e..2261b96474d 100644
--- a/solr/core/src/java/org/apache/solr/cli/LinkConfigTool.java
+++ b/solr/core/src/java/org/apache/solr/cli/LinkConfigTool.java
@@ -18,10 +18,10 @@
 package org.apache.solr.cli;
 
 import java.io.PrintStream;
-import java.util.List;
 import java.util.concurrent.TimeUnit;
 import org.apache.commons.cli.CommandLine;
 import org.apache.commons.cli.Option;
+import org.apache.commons.cli.Options;
 import org.apache.solr.client.solrj.impl.SolrZkClientTimeout;
 import org.apache.solr.cloud.ZkController;
 import org.apache.solr.common.cloud.SolrZkClient;
@@ -29,6 +29,24 @@
 /** Supports linking a configset to a collection */
 public class LinkConfigTool extends ToolBase {
 
+  private static final Option COLLECTION_NAME_OPTION =
+      Option.builder("c")
+          .longOpt("name")
+          .hasArg()
+          .argName("NAME")
+          .required()
+          .desc("Name of the collection to link.")
+          .build();
+
+  private static final Option CONF_NAME_OPTION =
+      Option.builder("n")
+          .longOpt("conf-name")
+          .hasArg()
+          .argName("NAME")
+          .required()
+          .desc("Configset name in ZooKeeper.")
+          .build();
+
   public LinkConfigTool() {
     this(CLIO.getOutStream());
   }
@@ -48,30 +66,18 @@ public String getUsage() {
   }
 
   @Override
-  public List<Option> getOptions() {
-    return List.of(
-        Option.builder("c")
-            .longOpt("name")
-            .argName("NAME")
-            .hasArg()
-            .required(true)
-            .desc("Name of the collection to link.")
-            .build(),
-        Option.builder("n")
-            .longOpt("conf-name")
-            .argName("NAME")
-            .hasArg()
-            .required(true)
-            .desc("Configset name in ZooKeeper.")
-            .build(),
-        SolrCLI.OPTION_ZKHOST);
+  public Options getOptions() {
+    return super.getOptions()
+        .addOption(COLLECTION_NAME_OPTION)
+        .addOption(CONF_NAME_OPTION)
+        .addOption(CommonCLIOptions.ZK_HOST_OPTION);
   }
 
   @Override
   public void runImpl(CommandLine cli) throws Exception {
 
-    String collection = cli.getOptionValue("name");
-    String confName = cli.getOptionValue("conf-name");
+    String collection = cli.getOptionValue(COLLECTION_NAME_OPTION);
+    String confName = cli.getOptionValue(CONF_NAME_OPTION);
     String zkHost = SolrCLI.getZkHost(cli);
 
     try (SolrZkClient zkClient =
diff --git a/solr/core/src/java/org/apache/solr/cli/PackageTool.java b/solr/core/src/java/org/apache/solr/cli/PackageTool.java
index 0089e896ad4..cb8bc6493ad 100644
--- a/solr/core/src/java/org/apache/solr/cli/PackageTool.java
+++ b/solr/core/src/java/org/apache/solr/cli/PackageTool.java
@@ -24,10 +24,10 @@
 import java.lang.invoke.MethodHandles;
 import java.nio.file.Files;
 import java.nio.file.Path;
-import java.util.List;
 import java.util.Map;
 import org.apache.commons.cli.CommandLine;
 import org.apache.commons.cli.Option;
+import org.apache.commons.cli.Options;
 import org.apache.logging.log4j.Level;
 import org.apache.logging.log4j.core.config.Configurator;
 import org.apache.lucene.util.SuppressForbidden;
@@ -49,6 +49,49 @@ public class PackageTool extends ToolBase {
 
   private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
 
+  private static final Option COLLECTIONS_OPTION =
+      Option.builder()
+          .longOpt("collections")
+          .hasArg()
+          .argName("COLLECTIONS")
+          .desc(
+              "Specifies that this action should affect plugins for the given collections only, excluding cluster level plugins.")
+          .build();
+
+  private static final Option CLUSTER_OPTION =
+      Option.builder()
+          .longOpt("cluster")
+          .desc("Specifies that this action should affect cluster-level plugins only.")
+          .build();
+
+  private static final Option PARAM_OPTION =
+      Option.builder()
+          .longOpt("param")
+          .hasArgs()
+          .argName("PARAMS")
+          .desc("List of parameters to be used with deploy command.")
+          .build();
+
+  private static final Option UPDATE_OPTION =
+      Option.builder()
+          .longOpt("update")
+          .desc("If a deployment is an update over a previous deployment.")
+          .build();
+
+  private static final Option COLLECTION_OPTION =
+      Option.builder("c")
+          .longOpt("collection")
+          .hasArg()
+          .argName("COLLECTION")
+          .desc("The collection to apply the package to, not required.")
+          .build();
+
+  private static final Option NO_PROMPT_OPTION =
+      Option.builder("y")
+          .longOpt("no-prompt")
+          .desc("Don't prompt for input; accept all default choices, defaults to false.")
+          .build();
+
   @SuppressForbidden(
       reason = "Need to turn off logging, and SLF4J doesn't seem to provide for a way.")
   public PackageTool() {
@@ -115,8 +158,8 @@ public void runImpl(CommandLine cli) throws Exception {
               }
               break;
             case "list-deployed":
-              if (cli.hasOption("collection")) {
-                String collection = cli.getOptionValue("collection");
+              if (cli.hasOption(COLLECTION_OPTION)) {
+                String collection = cli.getOptionValue(COLLECTION_OPTION);
                 Map<String, SolrPackageInstance> packages =
                     packageManager.getPackagesDeployed(collection);
                 printGreen("Packages deployed on " + collection + ":");
@@ -162,23 +205,23 @@ public void runImpl(CommandLine cli) throws Exception {
               }
             case "deploy":
               {
-                if (cli.hasOption("cluster") || cli.hasOption("collections")) {
+                if (cli.hasOption(CLUSTER_OPTION) || cli.hasOption(COLLECTIONS_OPTION)) {
                   Pair<String, String> parsedVersion = parsePackageVersion(cli.getArgList().get(1));
                   String packageName = parsedVersion.first();
                   String version = parsedVersion.second();
-                  boolean noPrompt = cli.hasOption("no-prompt");
-                  boolean isUpdate = cli.hasOption("update");
+                  boolean noPrompt = cli.hasOption(NO_PROMPT_OPTION);
+                  boolean isUpdate = cli.hasOption(UPDATE_OPTION);
                   String[] collections =
-                      cli.hasOption("collections")
+                      cli.hasOption(COLLECTIONS_OPTION)
                           ? PackageUtils.validateCollections(
-                              cli.getOptionValue("collections").split(","))
+                              cli.getOptionValue(COLLECTIONS_OPTION).split(","))
                           : new String[] {};
-                  String[] parameters = cli.getOptionValues("param");
+                  String[] parameters = cli.getOptionValues(PARAM_OPTION);
                   packageManager.deploy(
                       packageName,
                       version,
                       collections,
-                      cli.hasOption("cluster"),
+                      cli.hasOption(CLUSTER_OPTION),
                       parameters,
                       isUpdate,
                       noPrompt);
@@ -190,7 +233,7 @@ public void runImpl(CommandLine cli) throws Exception {
               }
             case "undeploy":
               {
-                if (cli.hasOption("cluster") || cli.hasOption("collections")) {
+                if (cli.hasOption(CLUSTER_OPTION) || cli.hasOption(COLLECTIONS_OPTION)) {
                   Pair<String, String> parsedVersion = parsePackageVersion(cli.getArgList().get(1));
                   if (parsedVersion.second() != null) {
                     throw new SolrException(
@@ -200,13 +243,12 @@ public void runImpl(CommandLine cli) throws Exception {
                   }
                   String packageName = parsedVersion.first();
                   String[] collections =
-                      cli.hasOption("collections")
+                      cli.hasOption(COLLECTIONS_OPTION)
                           ? PackageUtils.validateCollections(
-                              cli.getOptionValue("collections").split(","))
+                              cli.getOptionValue(COLLECTIONS_OPTION).split(","))
                           : new String[] {};
-                  packageManager.undeploy(packageName, collections, cli.hasOption("cluster"));
+                  packageManager.undeploy(packageName, collections, cli.hasOption(CLUSTER_OPTION));
                 } else {
-
                   printRed(
                       "Either specify --cluster to undeploy cluster level plugins or -collections <list-of-collections> to undeploy collection level plugins");
                 }
@@ -318,41 +360,15 @@ private Pair<String, String> parsePackageVersion(String arg) {
   }
 
   @Override
-  public List<Option> getOptions() {
-    return List.of(
-        Option.builder()
-            .longOpt("collections")
-            .hasArg()
-            .argName("COLLECTIONS")
-            .desc(
-                "Specifies that this action should affect plugins for the given collections only, excluding cluster level plugins.")
-            .build(),
-        Option.builder()
-            .longOpt("cluster")
-            .desc("Specifies that this action should affect cluster-level plugins only.")
-            .build(),
-        Option.builder()
-            .longOpt("param")
-            .hasArgs()
-            .argName("PARAMS")
-            .desc("List of parameters to be used with deploy command.")
-            .build(),
-        Option.builder()
-            .longOpt("update")
-            .desc("If a deployment is an update over a previous deployment.")
-            .build(),
-        Option.builder("c")
-            .longOpt("collection")
-            .hasArg()
-            .argName("COLLECTION")
-            .desc("The collection to apply the package to, not required.")
-            .build(),
-        Option.builder("y")
-            .longOpt("no-prompt")
-            .desc("Don't prompt for input; accept all default choices, defaults to false.")
-            .build(),
-        SolrCLI.OPTION_SOLRURL,
-        SolrCLI.OPTION_ZKHOST,
-        SolrCLI.OPTION_CREDENTIALS);
+  public Options getOptions() {
+    return super.getOptions()
+        .addOption(COLLECTIONS_OPTION)
+        .addOption(CLUSTER_OPTION)
+        .addOption(PARAM_OPTION)
+        .addOption(UPDATE_OPTION)
+        .addOption(COLLECTION_OPTION)
+        .addOption(NO_PROMPT_OPTION)
+        .addOption(CommonCLIOptions.CREDENTIALS_OPTION)
+        .addOptionGroup(getConnectionOptions());
   }
 }
diff --git a/solr/core/src/java/org/apache/solr/cli/PostLogsTool.java b/solr/core/src/java/org/apache/solr/cli/PostLogsTool.java
index 7be103ea923..e1efeb0b920 100644
--- a/solr/core/src/java/org/apache/solr/cli/PostLogsTool.java
+++ b/solr/core/src/java/org/apache/solr/cli/PostLogsTool.java
@@ -37,6 +37,7 @@
 import java.util.stream.Stream;
 import org.apache.commons.cli.CommandLine;
 import org.apache.commons.cli.Option;
+import org.apache.commons.cli.Options;
 import org.apache.solr.client.solrj.SolrClient;
 import org.apache.solr.client.solrj.impl.Http2SolrClient;
 import org.apache.solr.client.solrj.request.UpdateRequest;
@@ -48,6 +49,24 @@
 /** A command line tool for indexing Solr logs in the out-of-the-box log format. */
 public class PostLogsTool extends ToolBase {
 
+  private static final Option COLLECTION_NAME_OPTION =
+      Option.builder("c")
+          .longOpt("name")
+          .hasArg()
+          .argName("NAME")
+          .required()
+          .desc("Name of the collection.")
+          .build();
+
+  private static final Option ROOT_DIR_OPTION =
+      Option.builder()
+          .longOpt("rootdir")
+          .hasArg()
+          .argName("DIRECTORY")
+          .required()
+          .desc("All files found at or below the root directory will be indexed.")
+          .build();
+
   public PostLogsTool() {
     this(CLIO.getOutStream());
   }
@@ -62,38 +81,26 @@ public String getName() {
   }
 
   @Override
-  public List<Option> getOptions() {
-    return List.of(
-        Option.builder("c")
-            .longOpt("name")
-            .hasArg()
-            .required(true)
-            .argName("NAME")
-            .desc("Name of the collection.")
-            .build(),
-        Option.builder("rootdir")
-            .longOpt("rootdir")
-            .hasArg()
-            .argName("DIRECTORY")
-            .required(true)
-            .desc("All files found at or below the root directory will be indexed.")
-            .build(),
-        SolrCLI.OPTION_SOLRURL,
-        SolrCLI.OPTION_CREDENTIALS);
+  public Options getOptions() {
+    return super.getOptions()
+        .addOption(COLLECTION_NAME_OPTION)
+        .addOption(ROOT_DIR_OPTION)
+        .addOption(CommonCLIOptions.SOLR_URL_OPTION)
+        .addOption(CommonCLIOptions.CREDENTIALS_OPTION);
   }
 
   @Override
   public void runImpl(CommandLine cli) throws Exception {
     String url = null;
-    if (cli.hasOption("solr-url")) {
-      url = SolrCLI.normalizeSolrUrl(cli) + "/solr/" + cli.getOptionValue("name");
+    if (cli.hasOption(CommonCLIOptions.SOLR_URL_OPTION)) {
+      url = SolrCLI.normalizeSolrUrl(cli) + "/solr/" + cli.getOptionValue(COLLECTION_NAME_OPTION);
 
     } else {
       // Could be required arg, but maybe we want to support --zk-host option too?
       throw new IllegalArgumentException("Must specify --solr-url.");
     }
-    String rootDir = cli.getOptionValue("rootdir");
-    String credentials = cli.getOptionValue("credentials");
+    String rootDir = cli.getOptionValue(ROOT_DIR_OPTION);
+    String credentials = cli.getOptionValue(CommonCLIOptions.CREDENTIALS_OPTION);
     runCommand(url, rootDir, credentials);
   }
 
diff --git a/solr/core/src/java/org/apache/solr/cli/PostTool.java b/solr/core/src/java/org/apache/solr/cli/PostTool.java
index b888fdca589..0e50e619043 100644
--- a/solr/core/src/java/org/apache/solr/cli/PostTool.java
+++ b/solr/core/src/java/org/apache/solr/cli/PostTool.java
@@ -68,6 +68,7 @@
 import javax.xml.xpath.XPathFactory;
 import org.apache.commons.cli.CommandLine;
 import org.apache.commons.cli.Option;
+import org.apache.commons.cli.Options;
 import org.apache.commons.io.output.NullOutputStream;
 import org.apache.solr.client.api.util.SolrVersion;
 import org.apache.solr.client.solrj.SolrClient;
@@ -94,6 +95,93 @@ public class PostTool extends ToolBase {
   private static final int MAX_WEB_DEPTH = 10;
   public static final String DEFAULT_CONTENT_TYPE = "application/json";
 
+  private static final Option COLLECTION_NAME_OPTION =
+      Option.builder("c")
+          .longOpt("name")
+          .hasArg()
+          .argName("NAME")
+          .required()
+          .desc("Name of the collection.")
+          .build();
+
+  private static final Option SKIP_COMMIT_OPTION =
+      Option.builder()
+          .longOpt("skip-commit")
+          .desc("Do not 'commit', and thus changes won't be visible till a commit occurs.")
+          .build();
+
+  private static final Option OPTIMIZE_OPTION =
+      Option.builder("o")
+          .longOpt("optimize")
+          .desc("Issue an optimize at end of posting documents.")
+          .build();
+
+  private static final Option MODE_OPTION =
+      Option.builder()
+          .longOpt("mode")
+          .hasArg()
+          .argName("mode")
+          .desc(
+              "Which mode the Post tool is running in, 'files' crawls local directory, 'web' crawls website, 'args' processes input args, and 'stdin' reads a command from standard in. default: files.")
+          .build();
+
+  private static final Option RECURSIVE_OPTION =
+      Option.builder("r")
+          .longOpt("recursive")
+          .hasArg()
+          .argName("recursive")
+          .type(Integer.class)
+          .desc("For web crawl, how deep to go. default: 1")
+          .build();
+
+  private static final Option DELAY_OPTION =
+      Option.builder("d")
+          .longOpt("delay")
+          .hasArg()
+          .argName("delay")
+          .type(Integer.class)
+          .desc(
+              "If recursive then delay will be the wait time between posts.  default: 10 for web, 0 for files")
+          .build();
+
+  private static final Option TYPE_OPTION =
+      Option.builder("t")
+          .longOpt("type")
+          .hasArg()
+          .argName("content-type")
+          .desc("Specify a specific mimetype to use, such as application/json.")
+          .build();
+
+  private static final Option FILE_TYPES_OPTION =
+      Option.builder("ft")
+          .longOpt("filetypes")
+          .hasArg()
+          .argName("<type>[,<type>,...]")
+          .desc("default: " + DEFAULT_FILE_TYPES)
+          .build();
+
+  private static final Option PARAMS_OPTION =
+      Option.builder()
+          .longOpt("params")
+          .hasArg()
+          .argName("<key>=<value>[&<key>=<value>...]")
+          .desc("Values must be URL-encoded; these pass through to Solr update request.")
+          .build();
+
+  private static final Option FORMAT_OPTION =
+      Option.builder()
+          .longOpt("format")
+          .desc(
+              "sends application/json content as Solr commands to /update instead of /update/json/docs.")
+          .build();
+
+  private static final Option DRY_RUN_OPTION =
+      Option.builder()
+          .longOpt("dry-run")
+          .desc(
+              "Performs a dry run of the posting process without actually sending documents to Solr.  Only works with files mode.")
+          .build();
+
   // Input args
   int recursive = 0;
   int delay = 0;
@@ -169,128 +257,68 @@ public String getName() {
   }
 
   @Override
-  public List<Option> getOptions() {
-    return List.of(
-        Option.builder("c")
-            .longOpt("name")
-            .hasArg()
-            .required(true)
-            .argName("NAME")
-            .desc("Name of the collection.")
-            .build(),
-        Option.builder()
-            .longOpt("skip-commit")
-            .desc("Do not 'commit', and thus changes won't be visible till a commit occurs.")
-            .build(),
-        Option.builder("o")
-            .longOpt("optimize")
-            .desc("Issue an optimize at end of posting documents.")
-            .build(),
-        Option.builder()
-            .longOpt("mode")
-            .hasArg()
-            .argName("mode")
-            .desc(
-                "Which mode the Post tool is running in, 'files' crawls local directory, 'web' crawls website, 'args' processes input args, and 'stdin' reads a command from standard in. default: files.")
-            .build(),
-        Option.builder("r")
-            .longOpt("recursive")
-            .hasArg()
-            .argName("recursive")
-            .required(false)
-            .desc("For web crawl, how deep to go. default: 1")
-            .build(),
-        Option.builder()
-            .longOpt("delay")
-            .hasArg()
-            .argName("delay")
-            .required(false)
-            .desc(
-                "If recursive then delay will be the wait time between posts.  default: 10 for web, 0 for files")
-            .build(),
-        Option.builder("t")
-            .longOpt("type")
-            .hasArg()
-            .argName("content-type")
-            .required(false)
-            .desc("Specify a specific mimetype to use, such as application/json.")
-            .build(),
-        Option.builder("ft")
-            .longOpt("filetypes")
-            .hasArg()
-            .argName("<type>[,<type>,...]")
-            .required(false)
-            .desc("default: " + DEFAULT_FILE_TYPES)
-            .build(),
-        Option.builder()
-            .longOpt("params")
-            .hasArg()
-            .argName("<key>=<value>[&<key>=<value>...]")
-            .required(false)
-            .desc("Values must be URL-encoded; these pass through to Solr update request.")
-            .build(),
-        Option.builder()
-            .longOpt("format")
-            .required(false)
-            .desc(
-                "sends application/json content as Solr commands to /update instead of /update/json/docs.")
-            .build(),
-        Option.builder()
-            .longOpt("dry-run")
-            .required(false)
-            .desc(
-                "Performs a dry run of the posting process without actually sending documents to Solr.  Only works with files mode.")
-            .build(),
-        SolrCLI.OPTION_SOLRURL,
-        SolrCLI.OPTION_CREDENTIALS);
+  public Options getOptions() {
+    return super.getOptions()
+        .addOption(COLLECTION_NAME_OPTION)
+        .addOption(SKIP_COMMIT_OPTION)
+        .addOption(OPTIMIZE_OPTION)
+        .addOption(MODE_OPTION)
+        .addOption(RECURSIVE_OPTION)
+        .addOption(DELAY_OPTION)
+        .addOption(TYPE_OPTION)
+        .addOption(FILE_TYPES_OPTION)
+        .addOption(PARAMS_OPTION)
+        .addOption(FORMAT_OPTION)
+        .addOption(DRY_RUN_OPTION)
+        .addOption(CommonCLIOptions.SOLR_URL_OPTION)
+        .addOption(CommonCLIOptions.CREDENTIALS_OPTION);
   }
 
   @Override
   public void runImpl(CommandLine cli) throws Exception {
-    SolrCLI.raiseLogLevelUnlessVerbose(cli);
-
     solrUpdateUrl = null;
-    if (cli.hasOption("solr-url")) {
+    if (cli.hasOption(CommonCLIOptions.SOLR_URL_OPTION)) {
       String url =
-          SolrCLI.normalizeSolrUrl(cli) + "/solr/" + cli.getOptionValue("name") + "/update";
+          SolrCLI.normalizeSolrUrl(cli)
+              + "/solr/"
+              + cli.getOptionValue(COLLECTION_NAME_OPTION)
+              + "/update";
       solrUpdateUrl = new URI(url);
 
     } else {
-      String url = SolrCLI.getDefaultSolrUrl() + "/solr/" + cli.getOptionValue("name") + "/update";
+      String url =
+          SolrCLI.getDefaultSolrUrl()
+              + "/solr/"
+              + cli.getOptionValue(COLLECTION_NAME_OPTION)
+              + "/update";
       solrUpdateUrl = new URI(url);
     }
 
-    String mode = cli.getOptionValue("mode", DATA_MODE_FILES);
+    String mode = cli.getOptionValue(MODE_OPTION, DATA_MODE_FILES);
 
-    dryRun = cli.hasOption("dry-run");
+    dryRun = cli.hasOption(DRY_RUN_OPTION);
 
-    if (cli.hasOption("type")) {
-      type = cli.getOptionValue("type");
+    if (cli.hasOption(TYPE_OPTION)) {
+      type = cli.getOptionValue(TYPE_OPTION);
       // Turn off automatically looking up the mimetype in favour of what is passed in.
       auto = false;
     }
-    format = cli.hasOption("format") ? FORMAT_SOLR : ""; // i.e not solr formatted json commands
-
-    if (cli.hasOption("filetypes")) {
-      fileTypes = cli.getOptionValue("filetypes");
-    }
-
-    delay = (mode.equals((DATA_MODE_WEB)) ? 10 : 0);
-    if (cli.hasOption("delay")) {
-      delay = Integer.parseInt(cli.getOptionValue("delay"));
-    }
-
-    recursive = Integer.parseInt(cli.getOptionValue("recursive", "1"));
+    format =
+        cli.hasOption(FORMAT_OPTION) ? FORMAT_SOLR : ""; // i.e not solr formatted json commands
+    fileTypes = cli.getOptionValue(FILE_TYPES_OPTION, PostTool.DEFAULT_FILE_TYPES);
 
-    out = cli.hasOption(SolrCLI.OPTION_VERBOSE.getLongOpt()) ? CLIO.getOutStream() : null;
-    commit = !cli.hasOption("skip-commit");
-    optimize = cli.hasOption("optimize");
+    int defaultDelay = (mode.equals((DATA_MODE_WEB)) ? 10 : 0);
+    delay = cli.getParsedOptionValue(DELAY_OPTION, defaultDelay);
+    recursive = cli.getParsedOptionValue(RECURSIVE_OPTION, 1);
 
-    credentials = cli.getOptionValue(SolrCLI.OPTION_CREDENTIALS.getLongOpt());
+    out = isVerbose() ? CLIO.getOutStream() : null;
+    commit = !cli.hasOption(SKIP_COMMIT_OPTION);
+    optimize = cli.hasOption(OPTIMIZE_OPTION);
 
+    credentials = cli.getOptionValue(CommonCLIOptions.CREDENTIALS_OPTION);
     args = cli.getArgs();
 
-    params = cli.getOptionValue("params", "");
+    params = cli.getOptionValue(PARAMS_OPTION, "");
 
     execute(mode);
   }
diff --git a/solr/core/src/java/org/apache/solr/cli/RunExampleTool.java b/solr/core/src/java/org/apache/solr/cli/RunExampleTool.java
index 51f1d634703..99307cf74f0 100644
--- a/solr/core/src/java/org/apache/solr/cli/RunExampleTool.java
+++ b/solr/core/src/java/org/apache/solr/cli/RunExampleTool.java
@@ -28,7 +28,6 @@
 import java.util.Arrays;
 import java.util.Collections;
 import java.util.HashMap;
-import java.util.List;
 import java.util.Locale;
 import java.util.Map;
 import java.util.Optional;
@@ -37,6 +36,7 @@
 import java.util.concurrent.TimeUnit;
 import org.apache.commons.cli.CommandLine;
 import org.apache.commons.cli.Option;
+import org.apache.commons.cli.Options;
 import org.apache.commons.exec.DefaultExecuteResultHandler;
 import org.apache.commons.exec.DefaultExecutor;
 import org.apache.commons.exec.ExecuteException;
@@ -67,6 +67,99 @@ public class RunExampleTool extends ToolBase {
   private static final String PROMPT_NUMBER_TOO_LARGE =
       "%d is too large! " + PROMPT_FOR_NUMBER_IN_RANGE;
 
+  private static final Option NO_PROMPT_OPTION =
+      Option.builder("y")
+          .longOpt("no-prompt")
+          .desc(
+              "Don't prompt for input; accept all defaults when running examples that accept user input.")
+          .build();
+
+  private static final Option EXAMPLE_OPTION =
+      Option.builder("e")
+          .longOpt("example")
+          .hasArg()
+          .argName("NAME")
+          .required()
+          .desc("Name of the example to launch, one of: cloud, techproducts, schemaless, films.")
+          .build();
+
+  private static final Option SCRIPT_OPTION =
+      Option.builder()
+          .longOpt("script")
+          .hasArg()
+          .argName("PATH")
+          .desc("Path to the bin/solr script.")
+          .build();
+
+  private static final Option SERVER_DIR_OPTION =
+      Option.builder("d")
+          .longOpt("server-dir")
+          .hasArg()
+          .argName("DIR")
+          .required()
+          .desc("Path to the Solr server directory.")
+          .build();
+
+  private static final Option FORCE_OPTION =
+      Option.builder("f")
+          .longOpt("force")
+          .argName("FORCE")
+          .desc("Force option in case Solr is run as root.")
+          .build();
+
+  private static final Option EXAMPLE_DIR_OPTION =
+      Option.builder()
+          .longOpt("example-dir")
+          .hasArg()
+          .argName("DIR")
+          .desc(
+              "Path to the Solr example directory; if not provided, ${serverDir}/../example is expected to exist.")
+          .build();
+
+  private static final Option URL_SCHEME_OPTION =
+      Option.builder()
+          .longOpt("url-scheme")
+          .hasArg()
+          .argName("SCHEME")
+          .desc("Solr URL scheme: http or https, defaults to http if not specified.")
+          .build();
+
+  private static final Option PORT_OPTION =
+      Option.builder("p")
+          .longOpt("port")
+          .hasArg()
+          .argName("PORT")
+          .desc("Specify the port to start the Solr HTTP listener on; default is 8983.")
+          .build();
+
+  private static final Option HOST_OPTION =
+      Option.builder()
+          .longOpt("host")
+          .hasArg()
+          .argName("HOSTNAME")
+          .desc("Specify the hostname for this Solr instance.")
+          .build();
+
+  private static final Option USER_MANAGED_OPTION =
+      Option.builder().longOpt("user-managed").desc("Start Solr in User Managed mode.").build();
+
+  private static final Option MEMORY_OPTION =
+      Option.builder("m")
+          .longOpt("memory")
+          .hasArg()
+          .argName("MEM")
+          .desc(
+              "Sets the min (-Xms) and max (-Xmx) heap size for the JVM, such as: -m 4g results in: -Xms4g -Xmx4g; by default, this script sets the heap size to 512m.")
+          .build();
+
+  private static final Option JVM_OPTS_OPTION =
+      Option.builder()
+          .longOpt("jvm-opts")
+          .hasArg()
+          .argName("OPTS")
+          .desc("Additional options to be passed to the JVM when starting example Solr server(s).")
+          .build();
+
   protected InputStream userInput;
   protected Executor executor;
   protected String script;
@@ -91,105 +184,35 @@ public String getName() {
   }
 
   @Override
-  public List<Option> getOptions() {
-    return List.of(
-        Option.builder("y")
-            .longOpt("no-prompt")
-            .required(false)
-            .desc(
-                "Don't prompt for input; accept all defaults when running examples that accept user input.")
-            .build(),
-        Option.builder("e")
-            .longOpt("example")
-            .hasArg()
-            .argName("NAME")
-            .required(true)
-            .desc("Name of the example to launch, one of: cloud, techproducts, schemaless, films.")
-            .build(),
-        Option.builder()
-            .longOpt("script")
-            .hasArg()
-            .argName("PATH")
-            .required(false)
-            .desc("Path to the bin/solr script.")
-            .build(),
-        Option.builder()
-            .longOpt("server-dir")
-            .hasArg()
-            .argName("DIR")
-            .required(true)
-            .desc("Path to the Solr server directory.")
-            .build(),
-        Option.builder("f")
-            .longOpt("force")
-            .argName("FORCE")
-            .desc("Force option in case Solr is run as root.")
-            .build(),
-        Option.builder()
-            .longOpt("example-dir")
-            .hasArg()
-            .argName("DIR")
-            .required(false)
-            .desc(
-                "Path to the Solr example directory; if not provided, ${serverDir}/../example is expected to exist.")
-            .build(),
-        Option.builder()
-            .longOpt("url-scheme")
-            .hasArg()
-            .argName("SCHEME")
-            .required(false)
-            .desc("Solr URL scheme: http or https, defaults to http if not specified.")
-            .build(),
-        Option.builder("p")
-            .longOpt("port")
-            .hasArg()
-            .argName("PORT")
-            .required(false)
-            .desc("Specify the port to start the Solr HTTP listener on; default is 8983.")
-            .build(),
-        Option.builder()
-            .longOpt("host")
-            .hasArg()
-            .argName("HOSTNAME")
-            .required(false)
-            .desc("Specify the hostname for this Solr instance.")
-            .build(),
-        Option.builder()
-            .longOpt("user-managed")
-            .required(false)
-            .desc("Start Solr in User Managed mode.")
-            .build(),
-        Option.builder("m")
-            .longOpt("memory")
-            .hasArg()
-            .argName("MEM")
-            .required(false)
-            .desc(
-                "Sets the min (-Xms) and max (-Xmx) heap size for the JVM, such as: -m 4g results in: -Xms4g -Xmx4g; by default, this script sets the heap size to 512m.")
-            .build(),
-        Option.builder()
-            .longOpt("jvm-opts")
-            .hasArg()
-            .argName("OPTS")
-            .required(false)
-            .desc(
-                "Additional options to be passed to the JVM when starting example Solr server(s).")
-            .build(),
-        SolrCLI.OPTION_ZKHOST);
+  public Options getOptions() {
+    return super.getOptions()
+        .addOption(NO_PROMPT_OPTION)
+        .addOption(EXAMPLE_OPTION)
+        .addOption(SCRIPT_OPTION)
+        .addOption(SERVER_DIR_OPTION)
+        .addOption(FORCE_OPTION)
+        .addOption(EXAMPLE_DIR_OPTION)
+        .addOption(URL_SCHEME_OPTION)
+        .addOption(PORT_OPTION)
+        .addOption(HOST_OPTION)
+        .addOption(USER_MANAGED_OPTION)
+        .addOption(MEMORY_OPTION)
+        .addOption(JVM_OPTS_OPTION)
+        .addOption(CommonCLIOptions.ZK_HOST_OPTION);
   }
 
   @Override
   public void runImpl(CommandLine cli) throws Exception {
-    this.urlScheme = cli.getOptionValue("url-scheme", "http");
+    this.urlScheme = cli.getOptionValue(URL_SCHEME_OPTION, "http");
 
-    serverDir = new File(cli.getOptionValue("server-dir"));
+    serverDir = new File(cli.getOptionValue(SERVER_DIR_OPTION));
     if (!serverDir.isDirectory())
       throw new IllegalArgumentException(
           "Value of --server-dir option is invalid! "
               + serverDir.getAbsolutePath()
               + " is not a directory!");
 
-    script = cli.getOptionValue("script");
+    script = cli.getOptionValue(SCRIPT_OPTION);
     if (script != null) {
       if (!(new File(script)).isFile())
         throw new IllegalArgumentException(
@@ -210,8 +233,8 @@ public void runImpl(CommandLine cli) throws Exception {
     }
 
     exampleDir =
-        (cli.hasOption("example-dir"))
-            ? new File(cli.getOptionValue("example-dir"))
+        (cli.hasOption(EXAMPLE_DIR_OPTION))
+            ? new File(cli.getOptionValue(EXAMPLE_DIR_OPTION))
             : new File(serverDir.getParent(), "example");
     if (!exampleDir.isDirectory())
       throw new IllegalArgumentException(
@@ -227,7 +250,7 @@ public void runImpl(CommandLine cli) throws Exception {
             + "\nscript="
             + script);
 
-    String exampleType = cli.getOptionValue("example");
+    String exampleType = cli.getOptionValue(EXAMPLE_OPTION);
     if ("cloud".equals(exampleType)) {
       runCloudExample(cli);
     } else if ("techproducts".equals(exampleType)
@@ -248,11 +271,11 @@ protected void runExample(CommandLine cli, String exampleName) throws Exception
     String configSet =
         "techproducts".equals(exampleName) ? "sample_techproducts_configs" : "_default";
 
-    boolean isCloudMode = !cli.hasOption("user-managed");
-    String zkHost = cli.getOptionValue('z');
+    boolean isCloudMode = !cli.hasOption(USER_MANAGED_OPTION);
+    String zkHost = cli.getOptionValue(CommonCLIOptions.ZK_HOST_OPTION);
     int port =
         Integer.parseInt(
-            cli.getOptionValue('p', System.getenv().getOrDefault("SOLR_PORT", "8983")));
+            cli.getOptionValue(PORT_OPTION, System.getenv().getOrDefault("SOLR_PORT", "8983")));
     Map<String, Object> nodeStatus =
         startSolr(new File(exDir, "solr"), isCloudMode, cli, port, zkHost, 30);
 
@@ -264,7 +287,7 @@ protected void runExample(CommandLine cli, String exampleName) throws Exception
     boolean cloudMode = nodeStatus.get("cloud") != null;
     if (cloudMode) {
       if (SolrCLI.safeCheckCollectionExists(
-          solrUrl, collectionName, cli.getOptionValue(SolrCLI.OPTION_CREDENTIALS.getLongOpt()))) {
+          solrUrl, collectionName, cli.getOptionValue(CommonCLIOptions.CREDENTIALS_OPTION))) {
         alreadyExists = true;
         echo(
             "\nWARNING: Collection '"
@@ -274,7 +297,7 @@ protected void runExample(CommandLine cli, String exampleName) throws Exception
     } else {
       String coreName = collectionName;
       if (SolrCLI.safeCheckCoreExists(
-          solrUrl, coreName, cli.getOptionValue(SolrCLI.OPTION_CREDENTIALS.getLongOpt()))) {
+          solrUrl, coreName, cli.getOptionValue(CommonCLIOptions.CREDENTIALS_OPTION))) {
         alreadyExists = true;
         echo(
             "\nWARNING: Core '"
@@ -433,12 +456,12 @@ protected void runExample(CommandLine cli, String exampleName) throws Exception
 
   protected void runCloudExample(CommandLine cli) throws Exception {
 
-    boolean prompt = !cli.hasOption("no-prompt");
+    boolean prompt = !cli.hasOption(NO_PROMPT_OPTION);
     int numNodes = 2;
     int[] cloudPorts = new int[] {8983, 7574, 8984, 7575};
     int defaultPort =
         Integer.parseInt(
-            cli.getOptionValue('p', System.getenv().getOrDefault("SOLR_PORT", "8983")));
+            cli.getOptionValue(PORT_OPTION, System.getenv().getOrDefault("SOLR_PORT", "8983")));
     if (defaultPort != 8983) {
       // Override the old default port numbers if user has started the example overriding SOLR_PORT
       cloudPorts = new int[] {defaultPort, defaultPort + 1, defaultPort + 2, defaultPort + 3};
@@ -502,7 +525,7 @@ protected void runCloudExample(CommandLine cli) throws Exception {
     }
 
     // deal with extra args passed to the script to run the example
-    String zkHost = cli.getOptionValue('z');
+    String zkHost = cli.getOptionValue(CommonCLIOptions.ZK_HOST_OPTION);
 
     // start the first node (most likely with embedded ZK)
     Map<String, Object> nodeStatus =
@@ -539,7 +562,7 @@ protected void runCloudExample(CommandLine cli) throws Exception {
             readInput,
             prompt,
             solrUrl,
-            cli.getOptionValue(SolrCLI.OPTION_CREDENTIALS.getLongOpt()));
+            cli.getOptionValue(CommonCLIOptions.CREDENTIALS_OPTION));
 
     echo("\n\nSolrCloud example running, please visit: " + solrUrl + " \n");
   }
@@ -593,18 +616,17 @@ protected Map<String, Object> startSolr(
 
     String extraArgs = readExtraArgs(cli.getArgs());
 
-    String host = cli.getOptionValue("host");
-    String memory = cli.getOptionValue('m');
+    String host = cli.getOptionValue(HOST_OPTION);
+    String memory = cli.getOptionValue(MEMORY_OPTION);
 
     String hostArg = (host != null && !"localhost".equals(host)) ? " --host " + host : "";
     String zkHostArg = (zkHost != null) ? " -z " + zkHost : "";
     String memArg = (memory != null) ? " -m " + memory : "";
     String cloudModeArg = cloudMode ? "" : "--user-managed";
-    String forceArg = cli.hasOption("force") ? " --force" : "";
-    String verboseArg = verbose ? "--verbose" : "";
+    String forceArg = cli.hasOption(FORCE_OPTION) ? " --force" : "";
+    String verboseArg = isVerbose() ? "--verbose" : "";
 
-    String jvmOpts =
-        cli.hasOption("jvm-opts") ? cli.getOptionValue("jvm-opts") : cli.getOptionValue('a');
+    String jvmOpts = cli.getOptionValue(JVM_OPTS_OPTION);
     String jvmOptsArg = (jvmOpts != null) ? " --jvm-opts \"" + jvmOpts + "\"" : "";
 
     File cwd = new File(System.getProperty("user.dir"));
@@ -694,7 +716,7 @@ protected Map<String, Object> startSolr(
     if (code != 0) throw new Exception("Failed to start Solr using command: " + startCmd);
 
     return getNodeStatus(
-        solrUrl, cli.getOptionValue(SolrCLI.OPTION_CREDENTIALS.getLongOpt()), maxWaitSecs);
+        solrUrl, cli.getOptionValue(CommonCLIOptions.CREDENTIALS_OPTION), maxWaitSecs);
   }
 
   protected Map<String, Object> checkPortConflict(
@@ -985,7 +1007,7 @@ protected Integer promptForInt(
           }
 
         } catch (NumberFormatException nfe) {
-          if (verbose) echo(value + " is not a number!");
+          if (isVerbose()) echo(value + " is not a number!");
 
           if (min != null && max != null) {
             value =
diff --git a/solr/core/src/java/org/apache/solr/cli/SnapshotCreateTool.java b/solr/core/src/java/org/apache/solr/cli/SnapshotCreateTool.java
index d80dfbc61ea..236a7416832 100644
--- a/solr/core/src/java/org/apache/solr/cli/SnapshotCreateTool.java
+++ b/solr/core/src/java/org/apache/solr/cli/SnapshotCreateTool.java
@@ -17,9 +17,9 @@
 package org.apache.solr.cli;
 
 import java.io.PrintStream;
-import java.util.List;
 import org.apache.commons.cli.CommandLine;
 import org.apache.commons.cli.Option;
+import org.apache.commons.cli.Options;
 import org.apache.solr.client.solrj.SolrClient;
 import org.apache.solr.client.solrj.request.CollectionAdminRequest;
 import org.apache.solr.client.solrj.response.CollectionAdminResponse;
@@ -27,6 +27,24 @@
 /** Supports snapshot-create command in the bin/solr script. */
 public class SnapshotCreateTool extends ToolBase {
 
+  private static final Option COLLECTION_NAME_OPTION =
+      Option.builder("c")
+          .longOpt("name")
+          .hasArg()
+          .argName("NAME")
+          .required()
+          .desc("Name of collection to be snapshot.")
+          .build();
+
+  private static final Option SNAPSHOT_NAME_OPTION =
+      Option.builder()
+          .longOpt("snapshot-name")
+          .hasArg()
+          .argName("NAME")
+          .required()
+          .desc("Name of the snapshot to produce")
+          .build();
+
   public SnapshotCreateTool() {
     this(CLIO.getOutStream());
   }
@@ -41,33 +59,18 @@ public String getName() {
   }
 
   @Override
-  public List<Option> getOptions() {
-    return List.of(
-        SolrCLI.OPTION_ZKHOST,
-        SolrCLI.OPTION_SOLRURL,
-        Option.builder("c")
-            .longOpt("name")
-            .argName("NAME")
-            .hasArg()
-            .required(true)
-            .desc("Name of collection to be snapshot.")
-            .build(),
-        Option.builder()
-            .longOpt("snapshot-name")
-            .argName("NAME")
-            .hasArg()
-            .required(true)
-            .desc("Name of the snapshot to produce")
-            .build(),
-        SolrCLI.OPTION_CREDENTIALS);
+  public Options getOptions() {
+    return super.getOptions()
+        .addOption(COLLECTION_NAME_OPTION)
+        .addOption(SNAPSHOT_NAME_OPTION)
+        .addOption(CommonCLIOptions.CREDENTIALS_OPTION)
+        .addOptionGroup(getConnectionOptions());
   }
 
   @Override
   public void runImpl(CommandLine cli) throws Exception {
-    SolrCLI.raiseLogLevelUnlessVerbose(cli);
-
-    String snapshotName = cli.getOptionValue("snapshot-name");
-    String collectionName = cli.getOptionValue("name");
+    String snapshotName = cli.getOptionValue(SNAPSHOT_NAME_OPTION);
+    String collectionName = cli.getOptionValue(COLLECTION_NAME_OPTION);
     try (var solrClient = SolrCLI.getSolrClient(cli)) {
       createSnapshot(solrClient, collectionName, snapshotName);
     }
diff --git a/solr/core/src/java/org/apache/solr/cli/SnapshotDeleteTool.java b/solr/core/src/java/org/apache/solr/cli/SnapshotDeleteTool.java
index d172391183e..53fa8ad002a 100644
--- a/solr/core/src/java/org/apache/solr/cli/SnapshotDeleteTool.java
+++ b/solr/core/src/java/org/apache/solr/cli/SnapshotDeleteTool.java
@@ -17,9 +17,9 @@
 package org.apache.solr.cli;
 
 import java.io.PrintStream;
-import java.util.List;
 import org.apache.commons.cli.CommandLine;
 import org.apache.commons.cli.Option;
+import org.apache.commons.cli.Options;
 import org.apache.solr.client.solrj.SolrClient;
 import org.apache.solr.client.solrj.request.CollectionAdminRequest;
 import org.apache.solr.client.solrj.response.CollectionAdminResponse;
@@ -27,6 +27,24 @@
 /** Supports snapshot-delete command in the bin/solr script. */
 public class SnapshotDeleteTool extends ToolBase {
 
+  private static final Option COLLECTION_NAME_OPTION =
+      Option.builder("c")
+          .longOpt("name")
+          .hasArg()
+          .argName("NAME")
+          .required()
+          .desc("Name of collection to manage.")
+          .build();
+
+  private static final Option SNAPSHOT_NAME_OPTION =
+      Option.builder()
+          .longOpt("snapshot-name")
+          .hasArg()
+          .argName("NAME")
+          .required()
+          .desc("Name of the snapshot to delete")
+          .build();
+
   public SnapshotDeleteTool() {
     this(CLIO.getOutStream());
   }
@@ -41,33 +59,18 @@ public String getName() {
   }
 
   @Override
-  public List<Option> getOptions() {
-    return List.of(
-        SolrCLI.OPTION_ZKHOST,
-        SolrCLI.OPTION_SOLRURL,
-        Option.builder("c")
-            .longOpt("name")
-            .argName("NAME")
-            .hasArg()
-            .required(true)
-            .desc("Name of collection to manage.")
-            .build(),
-        Option.builder()
-            .longOpt("snapshot-name")
-            .argName("NAME")
-            .hasArg()
-            .required(true)
-            .desc("Name of the snapshot to delete")
-            .build(),
-        SolrCLI.OPTION_CREDENTIALS);
+  public Options getOptions() {
+    return super.getOptions()
+        .addOption(COLLECTION_NAME_OPTION)
+        .addOption(SNAPSHOT_NAME_OPTION)
+        .addOption(CommonCLIOptions.CREDENTIALS_OPTION)
+        .addOptionGroup(getConnectionOptions());
   }
 
   @Override
   public void runImpl(CommandLine cli) throws Exception {
-    SolrCLI.raiseLogLevelUnlessVerbose(cli);
-
-    String snapshotName = cli.getOptionValue("snapshot-name");
-    String collectionName = cli.getOptionValue("name");
+    String snapshotName = cli.getOptionValue(SNAPSHOT_NAME_OPTION);
+    String collectionName = cli.getOptionValue(COLLECTION_NAME_OPTION);
     try (var solrClient = SolrCLI.getSolrClient(cli)) {
       deleteSnapshot(solrClient, collectionName, snapshotName);
     }
diff --git a/solr/core/src/java/org/apache/solr/cli/SnapshotDescribeTool.java b/solr/core/src/java/org/apache/solr/cli/SnapshotDescribeTool.java
index a007b75f075..b76f36a7420 100644
--- a/solr/core/src/java/org/apache/solr/cli/SnapshotDescribeTool.java
+++ b/solr/core/src/java/org/apache/solr/cli/SnapshotDescribeTool.java
@@ -22,10 +22,10 @@
 import java.text.SimpleDateFormat;
 import java.util.ArrayList;
 import java.util.Collection;
-import java.util.List;
 import java.util.Locale;
 import org.apache.commons.cli.CommandLine;
 import org.apache.commons.cli.Option;
+import org.apache.commons.cli.Options;
 import org.apache.solr.client.solrj.SolrClient;
 import org.apache.solr.client.solrj.SolrServerException;
 import org.apache.solr.client.solrj.request.CollectionAdminRequest;
@@ -37,6 +37,24 @@
 /** Supports snapshot-describe command in the bin/solr script. */
 public class SnapshotDescribeTool extends ToolBase {
 
+  private static final Option COLLECTION_NAME_OPTION =
+      Option.builder("c")
+          .longOpt("name")
+          .hasArg()
+          .argName("NAME")
+          .required()
+          .desc("Name of collection to be snapshot.")
+          .build();
+
+  private static final Option SNAPSHOT_NAME_OPTION =
+      Option.builder()
+          .longOpt("snapshot-name")
+          .hasArg()
+          .argName("NAME")
+          .required()
+          .desc("Name of the snapshot to describe")
+          .build();
+
   public SnapshotDescribeTool() {
     this(CLIO.getOutStream());
   }
@@ -54,33 +72,18 @@ public String getName() {
   }
 
   @Override
-  public List<Option> getOptions() {
-    return List.of(
-        SolrCLI.OPTION_ZKHOST,
-        SolrCLI.OPTION_SOLRURL,
-        Option.builder("c")
-            .longOpt("name")
-            .argName("NAME")
-            .hasArg()
-            .required(true)
-            .desc("Name of collection to be snapshot.")
-            .build(),
-        Option.builder()
-            .longOpt("snapshot-name")
-            .argName("NAME")
-            .hasArg()
-            .required(true)
-            .desc("Name of the snapshot to describe")
-            .build(),
-        SolrCLI.OPTION_CREDENTIALS);
+  public Options getOptions() {
+    return super.getOptions()
+        .addOption(COLLECTION_NAME_OPTION)
+        .addOption(SNAPSHOT_NAME_OPTION)
+        .addOption(CommonCLIOptions.CREDENTIALS_OPTION)
+        .addOptionGroup(getConnectionOptions());
   }
 
   @Override
   public void runImpl(CommandLine cli) throws Exception {
-    SolrCLI.raiseLogLevelUnlessVerbose(cli);
-
-    String snapshotName = cli.getOptionValue("snapshot-name");
-    String collectionName = cli.getOptionValue("name");
+    String snapshotName = cli.getOptionValue(SNAPSHOT_NAME_OPTION);
+    String collectionName = cli.getOptionValue(COLLECTION_NAME_OPTION);
     try (var solrClient = SolrCLI.getSolrClient(cli)) {
       describeSnapshot(solrClient, collectionName, snapshotName);
     }
diff --git a/solr/core/src/java/org/apache/solr/cli/SnapshotExportTool.java b/solr/core/src/java/org/apache/solr/cli/SnapshotExportTool.java
index 8471ec79287..a51e91badb3 100644
--- a/solr/core/src/java/org/apache/solr/cli/SnapshotExportTool.java
+++ b/solr/core/src/java/org/apache/solr/cli/SnapshotExportTool.java
@@ -17,10 +17,10 @@
 package org.apache.solr.cli;
 
 import java.io.PrintStream;
-import java.util.List;
 import java.util.Optional;
 import org.apache.commons.cli.CommandLine;
 import org.apache.commons.cli.Option;
+import org.apache.commons.cli.Options;
 import org.apache.solr.client.solrj.SolrClient;
 import org.apache.solr.client.solrj.request.CollectionAdminRequest;
 import org.apache.solr.common.params.CollectionAdminParams;
@@ -28,6 +28,51 @@
 /** Supports snapshot-export command in the bin/solr script. */
 public class SnapshotExportTool extends ToolBase {
 
+  private static final Option COLLECTION_NAME_OPTION =
+      Option.builder("c")
+          .longOpt("name")
+          .hasArg()
+          .argName("NAME")
+          .required()
+          .desc("Name of collection to be snapshot.")
+          .build();
+
+  private static final Option SNAPSHOT_NAME_OPTION =
+      Option.builder()
+          .longOpt("snapshot-name")
+          .hasArg()
+          .argName("NAME")
+          .required()
+          .desc("Name of the snapshot to be exported.")
+          .build();
+
+  private static final Option DEST_DIR_OPTION =
+      Option.builder()
+          .longOpt("dest-dir")
+          .hasArg()
+          .argName("DIR")
+          .required()
+          .desc("Path of a temporary directory on local filesystem during snapshot export command.")
+          .build();
+
+  private static final Option BACKUP_REPO_NAME_OPTION =
+      Option.builder()
+          .longOpt("backup-repo-name")
+          .hasArg()
+          .argName("DIR")
+          .desc(
+              "Specifies name of the backup repository to be used during snapshot export preparation.")
+          .build();
+
+  private static final Option ASYNC_ID_OPTION =
+      Option.builder()
+          .longOpt("async-id")
+          .hasArg()
+          .argName("ID")
+          .desc(
+              "Specifies the async request identifier to be used during snapshot export preparation.")
+          .build();
+
   public SnapshotExportTool() {
     this(CLIO.getOutStream());
   }
@@ -42,60 +87,24 @@ public String getName() {
   }
 
   @Override
-  public List<Option> getOptions() {
-    return List.of(
-        SolrCLI.OPTION_ZKHOST,
-        SolrCLI.OPTION_SOLRURL,
-        Option.builder("c")
-            .longOpt("name")
-            .argName("NAME")
-            .hasArg()
-            .required(true)
-            .desc("Name of collection to be snapshot.")
-            .build(),
-        Option.builder()
-            .longOpt("snapshot-name")
-            .argName("NAME")
-            .hasArg()
-            .required(true)
-            .desc("Name of the snapshot to be exported.")
-            .build(),
-        Option.builder()
-            .longOpt("dest-dir")
-            .argName("DIR")
-            .hasArg()
-            .required(true)
-            .desc(
-                "Path of a temporary directory on local filesystem during snapshot export command.")
-            .build(),
-        Option.builder()
-            .longOpt("backup-repo-name")
-            .argName("DIR")
-            .hasArg()
-            .required(false)
-            .desc(
-                "Specifies name of the backup repository to be used during snapshot export preparation.")
-            .build(),
-        Option.builder()
-            .longOpt("async-id")
-            .argName("ID")
-            .hasArg()
-            .required(false)
-            .desc(
-                "Specifies the async request identifier to be used during snapshot export preparation.")
-            .build(),
-        SolrCLI.OPTION_CREDENTIALS);
+  public Options getOptions() {
+    return super.getOptions()
+        .addOption(COLLECTION_NAME_OPTION)
+        .addOption(SNAPSHOT_NAME_OPTION)
+        .addOption(DEST_DIR_OPTION)
+        .addOption(BACKUP_REPO_NAME_OPTION)
+        .addOption(ASYNC_ID_OPTION)
+        .addOption(CommonCLIOptions.CREDENTIALS_OPTION)
+        .addOptionGroup(getConnectionOptions());
   }
 
   @Override
   public void runImpl(CommandLine cli) throws Exception {
-    SolrCLI.raiseLogLevelUnlessVerbose(cli);
-
-    String snapshotName = cli.getOptionValue("snapshot-name");
-    String collectionName = cli.getOptionValue("name");
-    String destDir = cli.getOptionValue("dest-dir");
-    Optional<String> backupRepo = Optional.ofNullable(cli.getOptionValue("backup-repo-name"));
-    Optional<String> asyncReqId = Optional.ofNullable(cli.getOptionValue("async-id"));
+    String snapshotName = cli.getOptionValue(SNAPSHOT_NAME_OPTION);
+    String collectionName = cli.getOptionValue(COLLECTION_NAME_OPTION);
+    String destDir = cli.getOptionValue(DEST_DIR_OPTION);
+    Optional<String> backupRepo = Optional.ofNullable(cli.getOptionValue(BACKUP_REPO_NAME_OPTION));
+    Optional<String> asyncReqId = Optional.ofNullable(cli.getOptionValue(ASYNC_ID_OPTION));
 
     try (var solrClient = SolrCLI.getSolrClient(cli)) {
       exportSnapshot(solrClient, collectionName, snapshotName, destDir, backupRepo, asyncReqId);
diff --git a/solr/core/src/java/org/apache/solr/cli/SnapshotListTool.java b/solr/core/src/java/org/apache/solr/cli/SnapshotListTool.java
index a1db955b0c0..87219ce6021 100644
--- a/solr/core/src/java/org/apache/solr/cli/SnapshotListTool.java
+++ b/solr/core/src/java/org/apache/solr/cli/SnapshotListTool.java
@@ -17,9 +17,9 @@
 package org.apache.solr.cli;
 
 import java.io.PrintStream;
-import java.util.List;
 import org.apache.commons.cli.CommandLine;
 import org.apache.commons.cli.Option;
+import org.apache.commons.cli.Options;
 import org.apache.solr.client.solrj.SolrClient;
 import org.apache.solr.client.solrj.request.CollectionAdminRequest;
 import org.apache.solr.client.solrj.response.CollectionAdminResponse;
@@ -29,6 +29,15 @@
 /** Supports snapshot-list command in the bin/solr script. */
 public class SnapshotListTool extends ToolBase {
 
+  private static final Option COLLECTION_NAME_OPTION =
+      Option.builder("c")
+          .longOpt("name")
+          .hasArg()
+          .argName("NAME")
+          .required()
+          .desc("Name of collection to list snapshots for.")
+          .build();
+
   public SnapshotListTool() {
     this(CLIO.getOutStream());
   }
@@ -43,25 +52,16 @@ public String getName() {
   }
 
   @Override
-  public List<Option> getOptions() {
-    return List.of(
-        SolrCLI.OPTION_ZKHOST,
-        SolrCLI.OPTION_SOLRURL,
-        Option.builder("c")
-            .longOpt("name")
-            .argName("NAME")
-            .hasArg()
-            .required(true)
-            .desc("Name of collection to list snapshots for.")
-            .build(),
-        SolrCLI.OPTION_CREDENTIALS);
+  public Options getOptions() {
+    return super.getOptions()
+        .addOption(COLLECTION_NAME_OPTION)
+        .addOption(CommonCLIOptions.CREDENTIALS_OPTION)
+        .addOptionGroup(getConnectionOptions());
   }
 
   @Override
   public void runImpl(CommandLine cli) throws Exception {
-    SolrCLI.raiseLogLevelUnlessVerbose(cli);
-
-    String collectionName = cli.getOptionValue("name");
+    String collectionName = cli.getOptionValue(COLLECTION_NAME_OPTION);
     try (var solrClient = SolrCLI.getSolrClient(cli)) {
       listSnapshots(solrClient, collectionName);
     }
diff --git a/solr/core/src/java/org/apache/solr/cli/SolrCLI.java b/solr/core/src/java/org/apache/solr/cli/SolrCLI.java
index 9f762459fcb..00a97b4434c 100755
--- a/solr/core/src/java/org/apache/solr/cli/SolrCLI.java
+++ b/solr/core/src/java/org/apache/solr/cli/SolrCLI.java
@@ -68,7 +68,6 @@
 import org.apache.solr.common.util.ContentStreamBase;
 import org.apache.solr.common.util.EnvUtils;
 import org.apache.solr.common.util.NamedList;
-import org.apache.solr.util.StartupLoggingUtils;
 import org.apache.solr.util.configuration.SSLConfigurationsFactory;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
@@ -85,58 +84,6 @@ public class SolrCLI implements CLIO {
 
   private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
 
-  public static final String ZK_HOST = "localhost:9983";
-
-  public static final Option OPTION_ZKHOST =
-      Option.builder("z")
-          .longOpt("zk-host")
-          .argName("HOST")
-          .hasArg()
-          .required(false)
-          .desc(
-              "Zookeeper connection string; unnecessary if ZK_HOST is defined in solr.in.sh; otherwise, defaults to "
-                  + ZK_HOST
-                  + '.')
-          .build();
-  public static final Option OPTION_SOLRURL =
-      Option.builder("s")
-          .longOpt("solr-url")
-          .argName("HOST")
-          .hasArg()
-          .required(false)
-          .desc(
-              "Base Solr URL, which can be used to determine the zk-host if that's not known; defaults to: "
-                  + getDefaultSolrUrl()
-                  + '.')
-          .build();
-
-  public static final Option OPTION_VERBOSE =
-      Option.builder()
-          .longOpt("verbose")
-          .required(false)
-          .desc("Enable verbose command output.")
-          .build();
-
-  public static final Option OPTION_HELP =
-      Option.builder("h").longOpt("help").required(false).desc("Print this message.").build();
-
-  public static final Option OPTION_RECURSIVE =
-      Option.builder("r")
-          .longOpt("recursive")
-          .required(false)
-          .desc("Apply the command recursively.")
-          .build();
-
-  public static final Option OPTION_CREDENTIALS =
-      Option.builder("u")
-          .longOpt("credentials")
-          .argName("credentials")
-          .hasArg()
-          .required(false)
-          .desc(
-              "Credentials in the format username:password. Example: --credentials solr:SolrRocks")
-          .build();
-
   public static void exit(int exitStatus) {
     try {
       System.exit(exitStatus);
@@ -272,12 +219,6 @@ protected static void checkSslStoreSysProp(String solrInstallDir, String key) {
     }
   }
 
-  public static void raiseLogLevelUnlessVerbose(CommandLine cli) {
-    if (!cli.hasOption(SolrCLI.OPTION_VERBOSE.getLongOpt())) {
-      StartupLoggingUtils.changeLogLevel("WARN");
-    }
-  }
-
   // Creates an instance of the requested tool, using classpath scanning if necessary
   private static Tool newTool(String toolType) throws Exception {
     if ("healthcheck".equals(toolType)) return new HealthcheckTool();
@@ -322,21 +263,6 @@ private static Tool newTool(String toolType) throws Exception {
     throw new IllegalArgumentException(toolType + " is not a valid command!");
   }
 
-  /** Returns tool options for given tool, for usage display purposes. Hides deprecated options. */
-  public static Options getToolOptions(Tool tool) {
-    Options options = new Options();
-    options.addOption(OPTION_HELP);
-    options.addOption(OPTION_VERBOSE);
-
-    List<Option> toolOpts = tool.getOptions();
-    for (Option toolOpt : toolOpts) {
-      if (!toolOpt.isDeprecated()) {
-        options.addOption(toolOpt);
-      }
-    }
-    return options;
-  }
-
   /**
    * Returns the value of the option with the given name, or the value of the deprecated option. If
    * both values are null, then it returns the default value.
@@ -371,17 +297,7 @@ private static void deprecatedHandlerStdErr(Option o) {
 
   /** Parses the command-line arguments passed by the user. */
   public static CommandLine processCommandLineArgs(Tool tool, String[] args) {
-    List<Option> customOptions = tool.getOptions();
-    Options options = new Options();
-
-    options.addOption(OPTION_HELP);
-    options.addOption(OPTION_VERBOSE);
-
-    if (customOptions != null) {
-      for (Option customOption : customOptions) {
-        options.addOption(customOption);
-      }
-    }
+    Options options = tool.getOptions();
 
     CommandLine cli = null;
     try {
@@ -411,7 +327,7 @@ public static CommandLine processCommandLineArgs(Tool tool, String[] args) {
       }
     }
 
-    if (cli.hasOption("help")) {
+    if (cli.hasOption(CommonCLIOptions.HELP_OPTION)) {
       printToolHelp(tool);
       exit(0);
     }
@@ -422,16 +338,18 @@ public static CommandLine processCommandLineArgs(Tool tool, String[] args) {
   /** Prints tool help for a given tool */
   public static void printToolHelp(Tool tool) {
     HelpFormatter formatter = getFormatter();
-    Options optionsNoDeprecated = new Options();
-    SolrCLI.getToolOptions(tool).getOptions().stream()
+    Options nonDeprecatedOptions = new Options();
+
+    tool.getOptions().getOptions().stream()
         .filter(option -> !option.isDeprecated())
-        .forEach(optionsNoDeprecated::addOption);
+        .forEach(nonDeprecatedOptions::addOption);
+
     String usageString = tool.getUsage() == null ? "bin/solr " + tool.getName() : tool.getUsage();
     boolean autoGenerateUsage = tool.getUsage() == null;
     formatter.printHelp(
         usageString,
         "\n" + tool.getHeader(),
-        optionsNoDeprecated,
+        nonDeprecatedOptions,
         tool.getFooter(),
         autoGenerateUsage);
   }
@@ -543,13 +461,13 @@ public static SolrClient getSolrClient(String solrUrl, String credentials) {
 
   public static SolrClient getSolrClient(CommandLine cli, boolean barePath) throws Exception {
     String solrUrl = SolrCLI.normalizeSolrUrl(cli);
-    String credentials = cli.getOptionValue(SolrCLI.OPTION_CREDENTIALS.getLongOpt());
+    String credentials = cli.getOptionValue(CommonCLIOptions.CREDENTIALS_OPTION);
     return getSolrClient(solrUrl, credentials, barePath);
   }
 
   public static SolrClient getSolrClient(CommandLine cli) throws Exception {
     String solrUrl = SolrCLI.normalizeSolrUrl(cli);
-    String credentials = cli.getOptionValue(SolrCLI.OPTION_CREDENTIALS.getLongOpt());
+    String credentials = cli.getOptionValue(CommonCLIOptions.CREDENTIALS_OPTION);
     return getSolrClient(solrUrl, credentials, false);
   }
 
@@ -564,8 +482,6 @@ public static NamedList<Object> postJsonToSolr(
     return solrClient.request(req);
   }
 
-  public static final String DEFAULT_CONFIG_SET = "_default";
-
   private static final long MS_IN_MIN = 60 * 1000L;
   private static final long MS_IN_HOUR = MS_IN_MIN * 60L;
   private static final long MS_IN_DAY = MS_IN_HOUR * 24L;
@@ -672,10 +588,9 @@ public static String normalizeSolrUrl(String solrUrl, boolean logUrlFormatWarnin
    * ZooKeeper.
    */
   public static String normalizeSolrUrl(CommandLine cli) throws Exception {
-    String solrUrl = cli.getOptionValue("solr-url");
-
+    String solrUrl = cli.getOptionValue(CommonCLIOptions.SOLR_URL_OPTION);
     if (solrUrl == null) {
-      String zkHost = cli.getOptionValue("zk-host");
+      String zkHost = cli.getOptionValue(CommonCLIOptions.ZK_HOST_OPTION);
       if (zkHost == null) {
         solrUrl = SolrCLI.getDefaultSolrUrl();
         CLIO.err(
@@ -706,7 +621,7 @@ public static String normalizeSolrUrl(CommandLine cli) throws Exception {
    */
   public static String getZkHost(CommandLine cli) throws Exception {
 
-    String zkHost = cli.getOptionValue("zk-host");
+    String zkHost = cli.getOptionValue(CommonCLIOptions.ZK_HOST_OPTION);
     if (zkHost != null && !zkHost.isBlank()) {
       return zkHost;
     }
@@ -734,15 +649,11 @@ public static String getZkHost(CommandLine cli) throws Exception {
     return zkHost;
   }
 
-  public static SolrZkClient getSolrZkClient(CommandLine cli) throws Exception {
-    return getSolrZkClient(cli, getZkHost(cli));
-  }
-
   public static SolrZkClient getSolrZkClient(CommandLine cli, String zkHost) throws Exception {
     if (zkHost == null) {
       throw new IllegalStateException(
           "Solr at "
-              + cli.getOptionValue("solrUrl")
+              + cli.getOptionValue(CommonCLIOptions.SOLR_URL_OPTION)
               + " is running in standalone server mode, this command can only be used when running in SolrCloud mode.\n");
     }
     return new SolrZkClient.Builder()
diff --git a/solr/core/src/java/org/apache/solr/cli/StatusTool.java b/solr/core/src/java/org/apache/solr/cli/StatusTool.java
index 16ad16e5946..dcf0200e511 100644
--- a/solr/core/src/java/org/apache/solr/cli/StatusTool.java
+++ b/solr/core/src/java/org/apache/solr/cli/StatusTool.java
@@ -17,8 +17,6 @@
 
 package org.apache.solr.cli;
 
-import static org.apache.solr.cli.SolrCLI.OPTION_SOLRURL;
-
 import java.io.PrintStream;
 import java.net.URI;
 import java.net.URISyntaxException;
@@ -32,6 +30,8 @@
 import java.util.concurrent.TimeoutException;
 import org.apache.commons.cli.CommandLine;
 import org.apache.commons.cli.Option;
+import org.apache.commons.cli.OptionGroup;
+import org.apache.commons.cli.Options;
 import org.apache.solr.cli.SolrProcessManager.SolrProcess;
 import org.apache.solr.client.solrj.SolrClient;
 import org.apache.solr.client.solrj.SolrRequest;
@@ -49,65 +49,67 @@
  * <p>Get the status of a Solr server.
  */
 public class StatusTool extends ToolBase {
-  private final SolrProcessManager processMgr;
-
-  public StatusTool() {
-    this(CLIO.getOutStream());
-  }
 
-  public StatusTool(PrintStream stdout) {
-    super(stdout);
-    processMgr = new SolrProcessManager();
-  }
-
-  @Override
-  public String getName() {
-    return "status";
-  }
-
-  private static final Option OPTION_MAXWAITSECS =
+  private static final Option MAX_WAIT_SECS_OPTION =
       Option.builder()
           .longOpt("max-wait-secs")
-          .argName("SECS")
           .hasArg()
-          .required(false)
+          .argName("SECS")
+          .type(Integer.class)
           .deprecated() // Will make it a stealth option, not printed or complained about
           .desc("Wait up to the specified number of seconds to see Solr running.")
           .build();
 
-  public static final Option OPTION_PORT =
+  public static final Option PORT_OPTION =
       Option.builder("p")
           .longOpt("port")
-          .argName("PORT")
-          .required(false)
           .hasArg()
+          .argName("PORT")
+          .type(Integer.class)
           .desc("Port on localhost to check status for")
           .build();
 
-  public static final Option OPTION_SHORT =
+  public static final Option SHORT_OPTION =
       Option.builder()
           .longOpt("short")
           .argName("SHORT")
-          .required(false)
           .desc("Short format. Prints one URL per line for running instances")
           .build();
 
+  private final SolrProcessManager processMgr;
+
+  public StatusTool() {
+    this(CLIO.getOutStream());
+  }
+
+  public StatusTool(PrintStream stdout) {
+    super(stdout);
+    processMgr = new SolrProcessManager();
+  }
+
   @Override
-  public List<Option> getOptions() {
-    return List.of(OPTION_SOLRURL, OPTION_MAXWAITSECS, OPTION_PORT, OPTION_SHORT);
+  public String getName() {
+    return "status";
+  }
+
+  @Override
+  public Options getOptions() {
+    OptionGroup optionGroup = new OptionGroup();
+    optionGroup.addOption(PORT_OPTION);
+    optionGroup.addOption(CommonCLIOptions.SOLR_URL_OPTION);
+    return super.getOptions()
+        .addOption(MAX_WAIT_SECS_OPTION)
+        .addOption(SHORT_OPTION)
+        .addOption(CommonCLIOptions.CREDENTIALS_OPTION)
+        .addOptionGroup(optionGroup);
   }
 
   @Override
   public void runImpl(CommandLine cli) throws Exception {
-    String solrUrl = cli.getOptionValue(OPTION_SOLRURL);
-    Integer port =
-        cli.hasOption(OPTION_PORT) ? Integer.parseInt(cli.getOptionValue(OPTION_PORT)) : null;
-    boolean shortFormat = cli.hasOption(OPTION_SHORT);
-    int maxWaitSecs = Integer.parseInt(cli.getOptionValue("max-wait-secs", "0"));
-
-    if (port != null && solrUrl != null) {
-      throw new IllegalArgumentException("Only one of port or url can be specified");
-    }
+    String solrUrl = cli.getOptionValue(CommonCLIOptions.SOLR_URL_OPTION);
+    Integer port = cli.hasOption(PORT_OPTION) ? cli.getParsedOptionValue(PORT_OPTION) : null;
+    boolean shortFormat = cli.hasOption(SHORT_OPTION);
+    int maxWaitSecs = cli.getParsedOptionValue(MAX_WAIT_SECS_OPTION, 0);
 
     if (solrUrl != null) {
       if (!URLUtil.hasScheme(solrUrl)) {
@@ -165,8 +167,8 @@ public void runImpl(CommandLine cli) throws Exception {
   }
 
   private void printProcessStatus(SolrProcess process, CommandLine cli) throws Exception {
-    int maxWaitSecs = Integer.parseInt(cli.getOptionValue("max-wait-secs", "0"));
-    boolean shortFormat = cli.hasOption(OPTION_SHORT);
+    int maxWaitSecs = cli.getParsedOptionValue(MAX_WAIT_SECS_OPTION, 0);
+    boolean shortFormat = cli.hasOption(SHORT_OPTION);
     String pidUrl = process.getLocalUrl();
     if (shortFormat) {
       CLIO.out(pidUrl);
@@ -227,7 +229,7 @@ public boolean waitForSolrUp(String solrUrl, CommandLine cli, int maxWaitSecs) t
     try {
       waitToSeeSolrUp(
           solrUrl,
-          cli.getOptionValue(SolrCLI.OPTION_CREDENTIALS.getLongOpt()),
+          cli.getOptionValue(CommonCLIOptions.CREDENTIALS_OPTION),
           maxWaitSecs,
           TimeUnit.SECONDS);
       return true;
@@ -263,7 +265,7 @@ public String statusFromRunningSolr(String solrUrl, CommandLine cli) throws Exce
     try {
       CharArr arr = new CharArr();
       new JSONWriter(arr, 2)
-          .write(getStatus(solrUrl, cli.getOptionValue(SolrCLI.OPTION_CREDENTIALS.getLongOpt())));
+          .write(getStatus(solrUrl, cli.getOptionValue(CommonCLIOptions.CREDENTIALS_OPTION)));
       return arr.toString();
     } catch (Exception exc) {
       if (SolrCLI.exceptionIsAuthRelated(exc)) {
diff --git a/solr/core/src/java/org/apache/solr/cli/Tool.java b/solr/core/src/java/org/apache/solr/cli/Tool.java
index 0edde7cdae2..a95af511642 100644
--- a/solr/core/src/java/org/apache/solr/cli/Tool.java
+++ b/solr/core/src/java/org/apache/solr/cli/Tool.java
@@ -17,9 +17,8 @@
 
 package org.apache.solr.cli;
 
-import java.util.List;
 import org.apache.commons.cli.CommandLine;
-import org.apache.commons.cli.Option;
+import org.apache.commons.cli.Options;
 
 public interface Tool {
   /** Defines the interface to a Solr tool that can be run from this command-line app. */
@@ -50,7 +49,12 @@ default String getFooter() {
     return "\nPlease see the Reference Guide for more tools documentation: https://solr.apache.org/guide/solr/latest/deployment-guide/solr-control-script-reference.html";
   }
 
-  List<Option> getOptions();
+  /**
+   * Retrieve the {@link Options} supported by this tool.
+   *
+   * @return The {@link Options} this tool supports.
+   */
+  Options getOptions();
 
   int runTool(CommandLine cli) throws Exception;
 }
diff --git a/solr/core/src/java/org/apache/solr/cli/ToolBase.java b/solr/core/src/java/org/apache/solr/cli/ToolBase.java
index d35b63e1528..b0a0e1063a7 100644
--- a/solr/core/src/java/org/apache/solr/cli/ToolBase.java
+++ b/solr/core/src/java/org/apache/solr/cli/ToolBase.java
@@ -19,11 +19,14 @@
 
 import java.io.PrintStream;
 import org.apache.commons.cli.CommandLine;
+import org.apache.commons.cli.OptionGroup;
+import org.apache.commons.cli.Options;
+import org.apache.solr.util.StartupLoggingUtils;
 
 public abstract class ToolBase implements Tool {
 
+  private boolean verbose = false;
   protected PrintStream stdout;
-  protected boolean verbose = false;
 
   protected ToolBase() {
     this(CLIO.getOutStream());
@@ -48,9 +51,29 @@ protected void echo(final String msg) {
     stdout.println(msg);
   }
 
+  @Override
+  public Options getOptions() {
+    return new Options()
+        .addOption(CommonCLIOptions.HELP_OPTION)
+        .addOption(CommonCLIOptions.VERBOSE_OPTION);
+  }
+
+  /**
+   * Provides the two ways of connecting to Solr for CLI Tools
+   *
+   * @return OptionGroup validates that only one option is supplied by the caller.
+   */
+  public OptionGroup getConnectionOptions() {
+    OptionGroup optionGroup = new OptionGroup();
+    optionGroup.addOption(CommonCLIOptions.SOLR_URL_OPTION);
+    optionGroup.addOption(CommonCLIOptions.ZK_HOST_OPTION);
+    return optionGroup;
+  }
+
   @Override
   public int runTool(CommandLine cli) throws Exception {
-    verbose = cli.hasOption(SolrCLI.OPTION_VERBOSE.getLongOpt());
+    verbose = cli.hasOption(CommonCLIOptions.VERBOSE_OPTION);
+    raiseLogLevelUnlessVerbose();
 
     int toolExitStatus = 0;
     try {
@@ -71,5 +94,11 @@ public int runTool(CommandLine cli) throws Exception {
     return toolExitStatus;
   }
 
+  private void raiseLogLevelUnlessVerbose() {
+    if (!verbose) {
+      StartupLoggingUtils.changeLogLevel("WARN");
+    }
+  }
+
   public abstract void runImpl(CommandLine cli) throws Exception;
 }
diff --git a/solr/core/src/java/org/apache/solr/cli/UpdateACLTool.java b/solr/core/src/java/org/apache/solr/cli/UpdateACLTool.java
index e9a9c5b444e..996ada0a213 100644
--- a/solr/core/src/java/org/apache/solr/cli/UpdateACLTool.java
+++ b/solr/core/src/java/org/apache/solr/cli/UpdateACLTool.java
@@ -18,10 +18,9 @@
 package org.apache.solr.cli;
 
 import java.io.PrintStream;
-import java.util.List;
 import java.util.concurrent.TimeUnit;
 import org.apache.commons.cli.CommandLine;
-import org.apache.commons.cli.Option;
+import org.apache.commons.cli.Options;
 import org.apache.solr.client.solrj.impl.SolrZkClientTimeout;
 import org.apache.solr.cloud.ZkController;
 import org.apache.solr.common.cloud.SolrZkClient;
@@ -53,8 +52,8 @@ public String getUsage() {
   }
 
   @Override
-  public List<Option> getOptions() {
-    return List.of(SolrCLI.OPTION_ZKHOST);
+  public Options getOptions() {
+    return super.getOptions().addOption(CommonCLIOptions.ZK_HOST_OPTION);
   }
 
   @Override
diff --git a/solr/core/src/java/org/apache/solr/cli/VersionTool.java b/solr/core/src/java/org/apache/solr/cli/VersionTool.java
index f5e84e6c1f8..148bdf36a40 100644
--- a/solr/core/src/java/org/apache/solr/cli/VersionTool.java
+++ b/solr/core/src/java/org/apache/solr/cli/VersionTool.java
@@ -17,10 +17,7 @@
 
 package org.apache.solr.cli;
 
-import java.util.Collections;
-import java.util.List;
 import org.apache.commons.cli.CommandLine;
-import org.apache.commons.cli.Option;
 import org.apache.solr.client.api.util.SolrVersion;
 
 public class VersionTool extends ToolBase {
@@ -30,11 +27,6 @@ public String getName() {
     return "version";
   }
 
-  @Override
-  public List<Option> getOptions() {
-    return Collections.emptyList();
-  }
-
   @Override
   public void runImpl(CommandLine cli) throws Exception {
     CLIO.out("Solr version is: " + SolrVersion.LATEST);
diff --git a/solr/core/src/java/org/apache/solr/cli/ZkCpTool.java b/solr/core/src/java/org/apache/solr/cli/ZkCpTool.java
index 488618e2f41..b2045ef3541 100644
--- a/solr/core/src/java/org/apache/solr/cli/ZkCpTool.java
+++ b/solr/core/src/java/org/apache/solr/cli/ZkCpTool.java
@@ -23,12 +23,12 @@
 import java.lang.reflect.InvocationTargetException;
 import java.nio.file.Path;
 import java.nio.file.Paths;
-import java.util.List;
 import java.util.Locale;
 import java.util.Properties;
 import java.util.concurrent.TimeUnit;
 import org.apache.commons.cli.CommandLine;
 import org.apache.commons.cli.Option;
+import org.apache.commons.cli.Options;
 import org.apache.solr.client.solrj.impl.SolrZkClientTimeout;
 import org.apache.solr.common.SolrException;
 import org.apache.solr.common.cloud.SolrZkClient;
@@ -44,6 +44,14 @@
 public class ZkCpTool extends ToolBase {
   private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
 
+  private static final Option SOLR_HOME_OPTION =
+      Option.builder()
+          .longOpt("solr-home")
+          .hasArg()
+          .argName("DIR")
+          .desc("Required to look up configuration for compressing state.json.")
+          .build();
+
   public ZkCpTool() {
     this(CLIO.getOutStream());
   }
@@ -53,19 +61,12 @@ public ZkCpTool(PrintStream stdout) {
   }
 
   @Override
-  public List<Option> getOptions() {
-    return List.of(
-        Option.builder()
-            .longOpt("solr-home")
-            .argName("DIR")
-            .hasArg()
-            .required(false)
-            .desc("Required to look up configuration for compressing state.json.")
-            .build(),
-        SolrCLI.OPTION_RECURSIVE,
-        SolrCLI.OPTION_SOLRURL,
-        SolrCLI.OPTION_ZKHOST,
-        SolrCLI.OPTION_CREDENTIALS);
+  public Options getOptions() {
+    return super.getOptions()
+        .addOption(SOLR_HOME_OPTION)
+        .addOption(CommonCLIOptions.RECURSIVE_OPTION)
+        .addOption(CommonCLIOptions.CREDENTIALS_OPTION)
+        .addOptionGroup(getConnectionOptions());
   }
 
   @Override
@@ -123,13 +124,12 @@ public String getHeader() {
 
   @Override
   public void runImpl(CommandLine cli) throws Exception {
-    SolrCLI.raiseLogLevelUnlessVerbose(cli);
     String zkHost = SolrCLI.getZkHost(cli);
 
     echoIfVerbose("\nConnecting to ZooKeeper at " + zkHost + " ...");
     String src = cli.getArgs()[0];
     String dst = cli.getArgs()[1];
-    boolean recursive = cli.hasOption("recursive");
+    boolean recursive = cli.hasOption(CommonCLIOptions.RECURSIVE_OPTION);
     echo("Copying from '" + src + "' to '" + dst + "'. ZooKeeper at " + zkHost);
 
     boolean srcIsZk = src.toLowerCase(Locale.ROOT).startsWith("zk:");
@@ -158,7 +158,7 @@ public void runImpl(CommandLine cli) throws Exception {
     Compressor compressor = new ZLibCompressor();
 
     if (dstIsZk) {
-      String solrHome = cli.getOptionValue("solr-home");
+      String solrHome = cli.getOptionValue(SOLR_HOME_OPTION);
       if (StrUtils.isNullOrEmpty(solrHome)) {
         solrHome = System.getProperty("solr.home");
       }
diff --git a/solr/core/src/java/org/apache/solr/cli/ZkLsTool.java b/solr/core/src/java/org/apache/solr/cli/ZkLsTool.java
index cefb37b0e76..c8f1f55b266 100644
--- a/solr/core/src/java/org/apache/solr/cli/ZkLsTool.java
+++ b/solr/core/src/java/org/apache/solr/cli/ZkLsTool.java
@@ -18,9 +18,8 @@
 
 import java.io.PrintStream;
 import java.lang.invoke.MethodHandles;
-import java.util.List;
 import org.apache.commons.cli.CommandLine;
-import org.apache.commons.cli.Option;
+import org.apache.commons.cli.Options;
 import org.apache.solr.common.cloud.SolrZkClient;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
@@ -38,12 +37,11 @@ public ZkLsTool(PrintStream stdout) {
   }
 
   @Override
-  public List<Option> getOptions() {
-    return List.of(
-        SolrCLI.OPTION_RECURSIVE,
-        SolrCLI.OPTION_SOLRURL,
-        SolrCLI.OPTION_ZKHOST,
-        SolrCLI.OPTION_CREDENTIALS);
+  public Options getOptions() {
+    return super.getOptions()
+        .addOption(CommonCLIOptions.RECURSIVE_OPTION)
+        .addOption(CommonCLIOptions.CREDENTIALS_OPTION)
+        .addOptionGroup(getConnectionOptions());
   }
 
   @Override
@@ -59,14 +57,13 @@ public String getUsage() {
 
   @Override
   public void runImpl(CommandLine cli) throws Exception {
-    SolrCLI.raiseLogLevelUnlessVerbose(cli);
     String zkHost = SolrCLI.getZkHost(cli);
     String znode = cli.getArgs()[0];
 
     try (SolrZkClient zkClient = SolrCLI.getSolrZkClient(cli, zkHost)) {
       echoIfVerbose("\nConnecting to ZooKeeper at " + zkHost + " ...");
 
-      boolean recursive = cli.hasOption("recursive");
+      boolean recursive = cli.hasOption(CommonCLIOptions.RECURSIVE_OPTION);
       echoIfVerbose(
           "Getting listing for ZooKeeper node "
               + znode
diff --git a/solr/core/src/java/org/apache/solr/cli/ZkMkrootTool.java b/solr/core/src/java/org/apache/solr/cli/ZkMkrootTool.java
index 606586d1b4f..f2dc3703835 100644
--- a/solr/core/src/java/org/apache/solr/cli/ZkMkrootTool.java
+++ b/solr/core/src/java/org/apache/solr/cli/ZkMkrootTool.java
@@ -20,9 +20,9 @@
 
 import java.io.PrintStream;
 import java.lang.invoke.MethodHandles;
-import java.util.List;
 import org.apache.commons.cli.CommandLine;
 import org.apache.commons.cli.Option;
+import org.apache.commons.cli.Options;
 import org.apache.solr.common.cloud.SolrZkClient;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
@@ -31,6 +31,13 @@
 public class ZkMkrootTool extends ToolBase {
   private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
 
+  private static final Option FAIL_ON_EXISTS_OPTION =
+      Option.builder()
+          .longOpt("fail-on-exists")
+          .hasArg()
+          .desc("Raise an error if the root exists.  Defaults to false.")
+          .build();
+
   public ZkMkrootTool() {
     this(CLIO.getOutStream());
   }
@@ -40,17 +47,11 @@ public ZkMkrootTool(PrintStream stdout) {
   }
 
   @Override
-  public List<Option> getOptions() {
-    return List.of(
-        Option.builder()
-            .longOpt("fail-on-exists")
-            .hasArg()
-            .required(false)
-            .desc("Raise an error if the root exists.  Defaults to false.")
-            .build(),
-        SolrCLI.OPTION_SOLRURL,
-        SolrCLI.OPTION_ZKHOST,
-        SolrCLI.OPTION_CREDENTIALS);
+  public Options getOptions() {
+    return super.getOptions()
+        .addOption(FAIL_ON_EXISTS_OPTION)
+        .addOption(CommonCLIOptions.CREDENTIALS_OPTION)
+        .addOptionGroup(getConnectionOptions());
   }
 
   @Override
@@ -75,10 +76,9 @@ public String getHeader() {
 
   @Override
   public void runImpl(CommandLine cli) throws Exception {
-    SolrCLI.raiseLogLevelUnlessVerbose(cli);
     String zkHost = SolrCLI.getZkHost(cli);
     String znode = cli.getArgs()[0];
-    boolean failOnExists = cli.hasOption("fail-on-exists");
+    boolean failOnExists = cli.hasOption(FAIL_ON_EXISTS_OPTION);
 
     try (SolrZkClient zkClient = SolrCLI.getSolrZkClient(cli, zkHost)) {
       echoIfVerbose("\nConnecting to ZooKeeper at " + zkHost + " ...");
diff --git a/solr/core/src/java/org/apache/solr/cli/ZkMvTool.java b/solr/core/src/java/org/apache/solr/cli/ZkMvTool.java
index f49f1484c4b..41ca7d78822 100644
--- a/solr/core/src/java/org/apache/solr/cli/ZkMvTool.java
+++ b/solr/core/src/java/org/apache/solr/cli/ZkMvTool.java
@@ -20,10 +20,9 @@
 
 import java.io.PrintStream;
 import java.lang.invoke.MethodHandles;
-import java.util.List;
 import java.util.Locale;
 import org.apache.commons.cli.CommandLine;
-import org.apache.commons.cli.Option;
+import org.apache.commons.cli.Options;
 import org.apache.solr.client.solrj.SolrServerException;
 import org.apache.solr.common.cloud.SolrZkClient;
 import org.slf4j.Logger;
@@ -42,8 +41,10 @@ public ZkMvTool(PrintStream stdout) {
   }
 
   @Override
-  public List<Option> getOptions() {
-    return List.of(SolrCLI.OPTION_SOLRURL, SolrCLI.OPTION_ZKHOST, SolrCLI.OPTION_CREDENTIALS);
+  public Options getOptions() {
+    return super.getOptions()
+        .addOption(CommonCLIOptions.CREDENTIALS_OPTION)
+        .addOptionGroup(getConnectionOptions());
   }
 
   @Override
@@ -73,7 +74,6 @@ public String getHeader() {
 
   @Override
   public void runImpl(CommandLine cli) throws Exception {
-    SolrCLI.raiseLogLevelUnlessVerbose(cli);
     String zkHost = SolrCLI.getZkHost(cli);
 
     try (SolrZkClient zkClient = SolrCLI.getSolrZkClient(cli, zkHost)) {
diff --git a/solr/core/src/java/org/apache/solr/cli/ZkRmTool.java b/solr/core/src/java/org/apache/solr/cli/ZkRmTool.java
index 4f4e1bf62e6..cdb2e600334 100644
--- a/solr/core/src/java/org/apache/solr/cli/ZkRmTool.java
+++ b/solr/core/src/java/org/apache/solr/cli/ZkRmTool.java
@@ -18,10 +18,9 @@
 
 import java.io.PrintStream;
 import java.lang.invoke.MethodHandles;
-import java.util.List;
 import java.util.Locale;
 import org.apache.commons.cli.CommandLine;
-import org.apache.commons.cli.Option;
+import org.apache.commons.cli.Options;
 import org.apache.solr.client.solrj.SolrServerException;
 import org.apache.solr.common.cloud.SolrZkClient;
 import org.slf4j.Logger;
@@ -40,12 +39,11 @@ public ZkRmTool(PrintStream stdout) {
   }
 
   @Override
-  public List<Option> getOptions() {
-    return List.of(
-        SolrCLI.OPTION_RECURSIVE,
-        SolrCLI.OPTION_SOLRURL,
-        SolrCLI.OPTION_ZKHOST,
-        SolrCLI.OPTION_CREDENTIALS);
+  public Options getOptions() {
+    return super.getOptions()
+        .addOption(CommonCLIOptions.RECURSIVE_OPTION)
+        .addOption(CommonCLIOptions.CREDENTIALS_OPTION)
+        .addOptionGroup(getConnectionOptions());
   }
 
   @Override
@@ -60,11 +58,10 @@ public String getUsage() {
 
   @Override
   public void runImpl(CommandLine cli) throws Exception {
-    SolrCLI.raiseLogLevelUnlessVerbose(cli);
     String zkHost = SolrCLI.getZkHost(cli);
 
     String target = cli.getArgs()[0];
-    boolean recursive = cli.hasOption("recursive");
+    boolean recursive = cli.hasOption(CommonCLIOptions.RECURSIVE_OPTION);
 
     String znode = target;
     if (target.toLowerCase(Locale.ROOT).startsWith("zk:")) {
diff --git a/solr/core/src/java/org/apache/solr/cli/ZkToolHelp.java b/solr/core/src/java/org/apache/solr/cli/ZkToolHelp.java
index 14b541d3b33..525030dab11 100644
--- a/solr/core/src/java/org/apache/solr/cli/ZkToolHelp.java
+++ b/solr/core/src/java/org/apache/solr/cli/ZkToolHelp.java
@@ -19,13 +19,25 @@
 import static org.apache.solr.cli.SolrCLI.print;
 
 import java.io.PrintStream;
-import java.util.List;
 import org.apache.commons.cli.CommandLine;
 import org.apache.commons.cli.Option;
+import org.apache.commons.cli.Options;
 
 /** Supports zk help information in the bin/solr script. */
 public class ZkToolHelp extends ToolBase {
 
+  private static final Option PRINT_ZK_SUBCOMMAND_OPTION =
+      Option.builder()
+          .longOpt("print-zk-subcommand-usage")
+          .desc("Reminds user to prepend zk to invoke the command.")
+          .build();
+
+  private static final Option PRINT_LONG_ZK_USAGE_OPTION =
+      Option.builder()
+          .longOpt("print-long-zk-usage")
+          .desc("Invokes the detailed help for zk commands.")
+          .build();
+
   public ZkToolHelp() {
     this(CLIO.getOutStream());
   }
@@ -35,18 +47,10 @@ public ZkToolHelp(PrintStream stdout) {
   }
 
   @Override
-  public List<Option> getOptions() {
-    return List.of(
-        Option.builder()
-            .longOpt("print-zk-subcommand-usage")
-            .desc("Reminds user to prepend zk to invoke the command.")
-            .required(false)
-            .build(),
-        Option.builder()
-            .longOpt("print-long-zk-usage")
-            .required(false)
-            .desc("Invokes the detailed help for zk commands.")
-            .build());
+  public Options getOptions() {
+    return super.getOptions()
+        .addOption(PRINT_ZK_SUBCOMMAND_OPTION)
+        .addOption(PRINT_LONG_ZK_USAGE_OPTION);
   }
 
   @Override
@@ -56,16 +60,14 @@ public String getName() {
 
   @Override
   public void runImpl(CommandLine cli) throws Exception {
-    SolrCLI.raiseLogLevelUnlessVerbose(cli);
-
-    if (cli.hasOption("print-zk-subcommand-usage")) {
+    if (cli.hasOption(PRINT_ZK_SUBCOMMAND_OPTION)) {
       String scriptCommand = cli.getArgs()[0];
       print(
           "You must invoke this subcommand using the zk command.   bin/solr zk "
               + scriptCommand
               + ".");
     }
-    if (cli.hasOption("print-long-zk-usage")) {
+    if (cli.hasOption(PRINT_LONG_ZK_USAGE_OPTION)) {
       print("usage:");
       print(new ZkLsTool().getUsage());
       print(new ZkCpTool().getUsage());
diff --git a/solr/core/src/test/org/apache/solr/cli/SolrCLITest.java b/solr/core/src/test/org/apache/solr/cli/SolrCLITest.java
index db9138e3311..4b5290a1ff7 100644
--- a/solr/core/src/test/org/apache/solr/cli/SolrCLITest.java
+++ b/solr/core/src/test/org/apache/solr/cli/SolrCLITest.java
@@ -45,7 +45,4 @@ public void testUptime() {
     assertEquals(
         "106751991167 days, 7 hours, 12 minutes, 56 seconds", SolrCLI.uptime(Long.MAX_VALUE));
   }
-
-  @Test
-  public void testGetCredentials() {}
 }
diff --git a/solr/packaging/test/test_create.bats b/solr/packaging/test/test_create.bats
index a7a4c6b15d8..e9199358253 100644
--- a/solr/packaging/test/test_create.bats
+++ b/solr/packaging/test/test_create.bats
@@ -39,3 +39,8 @@ teardown() {
   run solr create -c COLL_NAME
   assert_output --partial "Created collection 'COLL_NAME'"
 }
+
+@test "multiple connection options are prevented" {
+  run solr create -c COLL_NAME2 --solr-url http://localhost:${SOLR_PORT} -z localhost:${ZK_PORT}
+  assert_output --partial "The option 'z' was specified but an option from this group has already been selected: 's'"
+}
diff --git a/solr/packaging/test/test_status.bats b/solr/packaging/test/test_status.bats
index d1f8a53bcb4..5399a0b9fd6 100644
--- a/solr/packaging/test/test_status.bats
+++ b/solr/packaging/test/test_status.bats
@@ -53,11 +53,14 @@ teardown() {
   solr stop
 }
 
+@test "multiple connection options are prevented" {
+  run solr status --port ${SOLR_PORT} --solr-url http://localhost:${SOLR_PORT}
+  assert_output --partial "The option 's' was specified but an option from this group has already been selected: 'p'"
+}
+
 @test "status with invalid --solr-url from user" {
-  solr start
   run solr status --solr-url http://invalidhost:${SOLR_PORT}
   assert_output --partial "Solr at http://invalidhost:${SOLR_PORT} not online"
-  solr stop
 }
 
 @test "status with --short format" {
diff --git a/solr/prometheus-exporter/src/java/org/apache/solr/prometheus/exporter/SolrExporter.java b/solr/prometheus-exporter/src/java/org/apache/solr/prometheus/exporter/SolrExporter.java
index ef24cda66a2..ebcff4e87a6 100644
--- a/solr/prometheus-exporter/src/java/org/apache/solr/prometheus/exporter/SolrExporter.java
+++ b/solr/prometheus-exporter/src/java/org/apache/solr/prometheus/exporter/SolrExporter.java
@@ -301,7 +301,7 @@ public static void main(String[] args) {
       }
 
       String configFile = commandLine.getOptionValue(configOption, DEFAULT_CONFIG);
-      int numberOfThreads = commandLine.getParsedOptionValue("num-threads", DEFAULT_NUM_THREADS);
+      int numberOfThreads = commandLine.getParsedOptionValue(numThreadsOption, DEFAULT_NUM_THREADS);
       int scrapeInterval =
           commandLine.getParsedOptionValue(scrapeIntervalOption, DEFAULT_SCRAPE_INTERVAL);
 

From 2550e7ee4f7d88d8a91755f03d6f9b4401699c90 Mon Sep 17 00:00:00 2001
From: Solr Bot <125606113+solrbot@users.noreply.github.com>
Date: Thu, 7 Nov 2024 00:25:56 +0100
Subject: [PATCH 30/32] chore(deps): update dependency
 com.github.spotbugs:spotbugs-annotations to v4.8.6 (#2129)

---
 versions.lock  | 2 +-
 versions.props | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/versions.lock b/versions.lock
index 11c37b2acd7..c865dea10e4 100644
--- a/versions.lock
+++ b/versions.lock
@@ -22,7 +22,7 @@ com.github.jai-imageio:jai-imageio-core:1.4.0 (1 constraints: 5c0ced01)
 com.github.junrar:junrar:7.5.3 (1 constraints: 660c1102)
 com.github.luben:zstd-jni:1.5.6-3 (1 constraints: 5f0d6136)
 com.github.openjson:openjson:1.0.12 (1 constraints: 8b0c6d0e)
-com.github.spotbugs:spotbugs-annotations:4.8.0 (1 constraints: 0e051736)
+com.github.spotbugs:spotbugs-annotations:4.8.6 (1 constraints: 14051d36)
 com.github.stephenc.jcip:jcip-annotations:1.0-1 (3 constraints: c71d2c87)
 com.github.virtuald:curvesapi:1.07 (1 constraints: 9e0ac7c0)
 com.google.android:annotations:4.1.1.4 (2 constraints: b918820a)
diff --git a/versions.props b/versions.props
index b9bb03803c3..d4689bb4003 100644
--- a/versions.props
+++ b/versions.props
@@ -7,7 +7,7 @@ com.carrotsearch:hppc=0.10.0
 com.cybozu.labs:langdetect=1.1-20120112
 com.fasterxml.jackson:jackson-bom=2.18.0
 com.github.ben-manes.caffeine:caffeine=3.1.8
-com.github.spotbugs:*=4.8.0
+com.github.spotbugs:*=4.8.6
 com.github.stephenc.jcip:jcip-annotations=1.0-1
 com.google.cloud:google-cloud-bom=0.224.0
 com.google.errorprone:*=2.23.0

From b657ec0c2176ef58c224147b71a91fa5a8267e51 Mon Sep 17 00:00:00 2001
From: Pierre Salagnac <82967811+psalagnac@users.noreply.github.com>
Date: Thu, 7 Nov 2024 08:14:40 +0100
Subject: [PATCH 31/32] SOLR-17453: Leverage waitForState() instead of busy
 waiting (#2737)

Leverage waitForState() instead of busy waiting in CREATE, MIGRATE, REINDEXCOLLECTION, MOVEREPLICA commands, and in some tests.
---
 solr/CHANGES.txt                              |   4 +-
 .../api/collections/CreateCollectionCmd.java  |  19 +-
 .../cloud/api/collections/MigrateCmd.java     |  39 ++-
 .../cloud/api/collections/MoveReplicaCmd.java |  45 ++-
 .../api/collections/ReindexCollectionCmd.java |  23 +-
 .../org/apache/solr/servlet/HttpSolrCall.java |  25 +-
 ...nkeyNothingIsSafeWithPullReplicasTest.java |   2 +-
 ...sMonkeySafeLeaderWithPullReplicasTest.java |   2 +-
 .../apache/solr/cloud/DeleteReplicaTest.java  |  12 +-
 .../apache/solr/cloud/HttpPartitionTest.java  |  39 +--
 .../LeaderFailureAfterFreshStartTest.java     |   8 +-
 .../solr/cloud/PeerSyncReplicationTest.java   |   8 +-
 .../apache/solr/cloud/TestPullReplica.java    |  21 +-
 .../cloud/TestPullReplicaErrorHandling.java   |  18 +-
 .../solr/cloud/TestRebalanceLeaders.java      | 312 +++++++-----------
 .../apache/solr/cloud/TestTlogReplica.java    |  65 ++--
 .../cloud/api/collections/ShardSplitTest.java |   4 +-
 ...aredFileSystemAutoReplicaFailoverTest.java |  71 ++--
 .../solr/hdfs/cloud/StressHdfsTest.java       |  12 +-
 ...tractChaosMonkeyNothingIsSafeTestBase.java |   2 +-
 ...AbstractChaosMonkeySafeLeaderTestBase.java |   2 +-
 .../solr/cloud/AbstractDistribZkTestBase.java |  87 ++---
 .../cloud/AbstractFullDistribZkTestBase.java  | 128 +++----
 .../AbstractUnloadDistributedZkTestBase.java  |  69 ++--
 24 files changed, 419 insertions(+), 598 deletions(-)

diff --git a/solr/CHANGES.txt b/solr/CHANGES.txt
index 3a03a54985a..907827a6fd0 100644
--- a/solr/CHANGES.txt
+++ b/solr/CHANGES.txt
@@ -143,7 +143,7 @@ Improvements
   when PKI is used between nodes. (Jason Gerlowski)
 
 * SOLR-17383: Resolved overlapping arguments in the Solr CLI.  Removed duplicative but differing arguments,
-  consolidated use of short form arguments -v to not have differing meanings based on tool.  Provide deprecation warning 
+  consolidated use of short form arguments -v to not have differing meanings based on tool.  Provide deprecation warning
   in command line when deprecated arguments are used. (Eric Pugh, Christos Malliaridis)
 
 * SOLR-17256: Deprecate SolrRequest `setBasePath` and `getBasePath` methods.  SolrJ users wishing to temporarily
@@ -181,6 +181,8 @@ Optimizations
 
 * SOLR-16503: Switched from HTTP1 to HTTP2 in SolrClientCloudManager by replacing CloudLegacySolrClient with CloudHttp2SolrClient. (Sanjay Dutt, David Smiley)
 
+* SOLR-17453: Leverage waitForState() instead of busy waiting in CREATE, MIGRATE, REINDEXCOLLECTION, MOVEREPLICA commands, and in some tests. (Pierre Salagnac)
+
 Bug Fixes
 ---------------------
 * SOLR-12429: Uploading a configset with a symbolic link produces a IOException. Now a error message to user generated instead. (Eric Pugh)
diff --git a/solr/core/src/java/org/apache/solr/cloud/api/collections/CreateCollectionCmd.java b/solr/core/src/java/org/apache/solr/cloud/api/collections/CreateCollectionCmd.java
index ccfd4a1e5e2..5592303580a 100644
--- a/solr/core/src/java/org/apache/solr/cloud/api/collections/CreateCollectionCmd.java
+++ b/solr/core/src/java/org/apache/solr/cloud/api/collections/CreateCollectionCmd.java
@@ -36,9 +36,11 @@
 import java.util.List;
 import java.util.Map;
 import java.util.NoSuchElementException;
+import java.util.Objects;
 import java.util.Properties;
 import java.util.concurrent.ConcurrentHashMap;
 import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
 import org.apache.solr.client.solrj.cloud.AlreadyExistsException;
 import org.apache.solr.client.solrj.cloud.BadVersionException;
 import org.apache.solr.client.solrj.cloud.DelegatingCloudManager;
@@ -221,24 +223,19 @@ public void call(ClusterState clusterState, ZkNodeProps message, NamedList<Objec
         }
 
         // wait for a while until we see the collection
-        TimeOut waitUntil =
-            new TimeOut(30, TimeUnit.SECONDS, ccc.getSolrCloudManager().getTimeSource());
-        boolean created = false;
-        while (!waitUntil.hasTimedOut()) {
-          waitUntil.sleep(100);
-          created = ccc.getSolrCloudManager().getClusterState().hasCollection(collectionName);
-          if (created) break;
-        }
-        if (!created) {
+        try {
+          newColl =
+              zkStateReader.waitForState(collectionName, 30, TimeUnit.SECONDS, Objects::nonNull);
+        } catch (TimeoutException e) {
           throw new SolrException(
               SolrException.ErrorCode.SERVER_ERROR,
-              "Could not fully create collection: " + collectionName);
+              "Could not fully create collection: " + collectionName,
+              e);
         }
 
         // refresh cluster state (value read below comes from Zookeeper watch firing following the
         // update done previously, be it by Overseer or by this thread when updates are distributed)
         clusterState = ccc.getSolrCloudManager().getClusterState();
-        newColl = clusterState.getCollection(collectionName);
       }
 
       final List<ReplicaPosition> replicaPositions;
diff --git a/solr/core/src/java/org/apache/solr/cloud/api/collections/MigrateCmd.java b/solr/core/src/java/org/apache/solr/cloud/api/collections/MigrateCmd.java
index cdcb3c27449..95d1eb2ac11 100644
--- a/solr/core/src/java/org/apache/solr/cloud/api/collections/MigrateCmd.java
+++ b/solr/core/src/java/org/apache/solr/cloud/api/collections/MigrateCmd.java
@@ -31,6 +31,7 @@
 import java.util.HashMap;
 import java.util.Map;
 import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
 import org.apache.solr.client.solrj.request.CoreAdminRequest;
 import org.apache.solr.cloud.DistributedClusterStateUpdater;
 import org.apache.solr.cloud.Overseer;
@@ -52,7 +53,6 @@
 import org.apache.solr.common.util.NamedList;
 import org.apache.solr.common.util.Utils;
 import org.apache.solr.handler.component.ShardHandler;
-import org.apache.solr.util.TimeOut;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
@@ -272,27 +272,26 @@ private void migrateKey(
 
     // wait for a while until we see the new rule
     log.info("Waiting to see routing rule updated in clusterstate");
-    TimeOut waitUntil =
-        new TimeOut(60, TimeUnit.SECONDS, ccc.getSolrCloudManager().getTimeSource());
-    boolean added = false;
-    while (!waitUntil.hasTimedOut()) {
-      waitUntil.sleep(100);
-      sourceCollection = zkStateReader.getClusterState().getCollection(sourceCollection.getName());
-      sourceSlice = sourceCollection.getSlice(sourceSlice.getName());
-      Map<String, RoutingRule> rules = sourceSlice.getRoutingRules();
-      if (rules != null) {
-        RoutingRule rule = rules.get(sourceRouter.getRouteKeyNoSuffix(splitKey) + "!");
-        if (rule != null && rule.getRouteRanges().contains(splitRange)) {
-          added = true;
-          break;
-        }
-      }
-    }
-    if (!added) {
+
+    try {
+      sourceCollection =
+          zkStateReader.waitForState(
+              sourceCollection.getName(),
+              60,
+              TimeUnit.SECONDS,
+              c -> {
+                Slice s = c.getSlice(sourceSlice.getName());
+                Map<String, RoutingRule> rules = s.getRoutingRules();
+                if (rules != null) {
+                  RoutingRule rule = rules.get(sourceRouter.getRouteKeyNoSuffix(splitKey) + "!");
+                  return rule != null && rule.getRouteRanges().contains(splitRange);
+                }
+                return false;
+              });
+    } catch (TimeoutException e) {
       throw new SolrException(
-          SolrException.ErrorCode.SERVER_ERROR, "Could not add routing rule: " + m);
+          SolrException.ErrorCode.SERVER_ERROR, "Could not add routing rule: " + m, e);
     }
-
     log.info("Routing rule added successfully");
 
     // Create temp core on source shard
diff --git a/solr/core/src/java/org/apache/solr/cloud/api/collections/MoveReplicaCmd.java b/solr/core/src/java/org/apache/solr/cloud/api/collections/MoveReplicaCmd.java
index cd5097993b4..48f065f537e 100644
--- a/solr/core/src/java/org/apache/solr/cloud/api/collections/MoveReplicaCmd.java
+++ b/solr/core/src/java/org/apache/solr/cloud/api/collections/MoveReplicaCmd.java
@@ -33,6 +33,7 @@
 import java.util.List;
 import java.util.Locale;
 import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
 import org.apache.solr.cloud.ActiveReplicaWatcher;
 import org.apache.solr.common.SolrCloseableLatch;
 import org.apache.solr.common.SolrException;
@@ -46,7 +47,6 @@
 import org.apache.solr.common.params.CoreAdminParams;
 import org.apache.solr.common.util.NamedList;
 import org.apache.solr.common.util.Utils;
-import org.apache.solr.util.TimeOut;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
@@ -161,9 +161,7 @@ private void moveReplica(
           dataDir.toString(),
           targetNode,
           async,
-          coll,
           replica,
-          slice,
           timeout,
           waitForFinalState);
     } else {
@@ -187,9 +185,7 @@ private void moveHdfsReplica(
       String dataDir,
       String targetNode,
       String async,
-      DocCollection coll,
       Replica replica,
-      Slice slice,
       int timeout,
       boolean waitForFinalState)
       throws Exception {
@@ -198,8 +194,8 @@ private void moveHdfsReplica(
       skipCreateReplicaInClusterState = "false";
       ZkNodeProps removeReplicasProps =
           new ZkNodeProps(
-              COLLECTION_PROP, coll.getName(),
-              SHARD_ID_PROP, slice.getName(),
+              COLLECTION_PROP, replica.getCollection(),
+              SHARD_ID_PROP, replica.getShard(),
               REPLICA_PROP, replica.getName());
       removeReplicasProps.getProperties().put(CoreAdminParams.DELETE_DATA_DIR, false);
       removeReplicasProps.getProperties().put(CoreAdminParams.DELETE_INDEX, false);
@@ -217,8 +213,8 @@ private void moveHdfsReplica(
             String.format(
                 Locale.ROOT,
                 "Failed to cleanup replica collection=%s shard=%s name=%s, failure=%s",
-                coll.getName(),
-                slice.getName(),
+                replica.getCollection(),
+                replica.getShard(),
                 replica.getName(),
                 deleteResult.get("failure"));
         log.warn(errorString);
@@ -226,17 +222,14 @@ private void moveHdfsReplica(
         return;
       }
 
-      TimeOut timeOut =
-          new TimeOut(20L, TimeUnit.SECONDS, ccc.getSolrCloudManager().getTimeSource());
-      while (!timeOut.hasTimedOut()) {
-        coll = ccc.getZkStateReader().getClusterState().getCollection(coll.getName());
-        if (coll.getReplica(replica.getName()) != null) {
-          timeOut.sleep(100);
-        } else {
-          break;
-        }
-      }
-      if (timeOut.hasTimedOut()) {
+      try {
+        ccc.getZkStateReader()
+            .waitForState(
+                replica.getCollection(),
+                20L,
+                TimeUnit.SECONDS,
+                c -> c.getReplica(replica.getName()) != null);
+      } catch (TimeoutException e) {
         results.add("failure", "Still see deleted replica in clusterstate!");
         return;
       }
@@ -246,9 +239,9 @@ private void moveHdfsReplica(
     ZkNodeProps addReplicasProps =
         new ZkNodeProps(
             COLLECTION_PROP,
-            coll.getName(),
+            replica.getCollection(),
             SHARD_ID_PROP,
-            slice.getName(),
+            replica.getShard(),
             CoreAdminParams.NODE,
             targetNode,
             CoreAdminParams.CORE_NODE_NAME,
@@ -277,8 +270,8 @@ private void moveHdfsReplica(
           String.format(
               Locale.ROOT,
               "Failed to create replica for collection=%s shard=%s" + " on node=%s, failure=%s",
-              coll.getName(),
-              slice.getName(),
+              replica.getCollection(),
+              replica.getShard(),
               targetNode,
               addResult.get("failure"));
       results.add("failure", errorString);
@@ -302,8 +295,8 @@ private void moveHdfsReplica(
           String.format(
               Locale.ROOT,
               "Failed to create replica for collection=%s shard=%s" + " on node=%s, failure=%s",
-              coll.getName(),
-              slice.getName(),
+              replica.getCollection(),
+              replica.getShard(),
               targetNode,
               addResult.get("failure"));
       log.warn(errorString);
diff --git a/solr/core/src/java/org/apache/solr/cloud/api/collections/ReindexCollectionCmd.java b/solr/core/src/java/org/apache/solr/cloud/api/collections/ReindexCollectionCmd.java
index 91299b3c259..15324ec61aa 100644
--- a/solr/core/src/java/org/apache/solr/cloud/api/collections/ReindexCollectionCmd.java
+++ b/solr/core/src/java/org/apache/solr/cloud/api/collections/ReindexCollectionCmd.java
@@ -25,8 +25,10 @@
 import java.util.List;
 import java.util.Locale;
 import java.util.Map;
+import java.util.Objects;
 import java.util.TreeMap;
 import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
 import java.util.concurrent.atomic.AtomicInteger;
 import java.util.function.Function;
 import java.util.stream.Collectors;
@@ -360,22 +362,17 @@ public void call(ClusterState clusterState, ZkNodeProps message, NamedList<Objec
       CollectionHandlingUtils.checkResults(
           "creating checkpoint collection " + chkCollection, cmdResults, true);
       // wait for a while until we see both collections
-      TimeOut waitUntil =
-          new TimeOut(30, TimeUnit.SECONDS, ccc.getSolrCloudManager().getTimeSource());
-      boolean created = false;
-      while (!waitUntil.hasTimedOut()) {
-        waitUntil.sleep(100);
-        // this also refreshes our local var clusterState
-        clusterState = ccc.getSolrCloudManager().getClusterState();
-        created =
-            clusterState.hasCollection(targetCollection)
-                && clusterState.hasCollection(chkCollection);
-        if (created) break;
-      }
-      if (!created) {
+      try {
+        for (String col : List.of(targetCollection, chkCollection)) {
+          ccc.getZkStateReader().waitForState(col, 30, TimeUnit.SECONDS, Objects::nonNull);
+        }
+      } catch (TimeoutException e) {
         throw new SolrException(
             SolrException.ErrorCode.SERVER_ERROR, "Could not fully create temporary collection(s)");
       }
+
+      clusterState = ccc.getSolrCloudManager().getClusterState();
+
       if (maybeAbort(collection)) {
         aborted = true;
         return;
diff --git a/solr/core/src/java/org/apache/solr/servlet/HttpSolrCall.java b/solr/core/src/java/org/apache/solr/servlet/HttpSolrCall.java
index cc645476aef..99291fc6ad0 100644
--- a/solr/core/src/java/org/apache/solr/servlet/HttpSolrCall.java
+++ b/solr/core/src/java/org/apache/solr/servlet/HttpSolrCall.java
@@ -53,6 +53,7 @@
 import java.util.Random;
 import java.util.Set;
 import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
 import java.util.function.Supplier;
 import javax.servlet.http.HttpServletRequest;
 import javax.servlet.http.HttpServletResponse;
@@ -96,7 +97,6 @@
 import org.apache.solr.common.util.SimpleOrderedMap;
 import org.apache.solr.common.util.StrUtils;
 import org.apache.solr.common.util.SuppressForbidden;
-import org.apache.solr.common.util.TimeSource;
 import org.apache.solr.common.util.Utils;
 import org.apache.solr.common.util.ValidatingJsonMap;
 import org.apache.solr.core.CoreContainer;
@@ -125,7 +125,6 @@
 import org.apache.solr.servlet.cache.Method;
 import org.apache.solr.update.processor.DistributingUpdateProcessorFactory;
 import org.apache.solr.util.RTimerTree;
-import org.apache.solr.util.TimeOut;
 import org.apache.solr.util.tracing.TraceUtils;
 import org.apache.zookeeper.KeeperException;
 import org.slf4j.Logger;
@@ -388,18 +387,16 @@ protected void autoCreateSystemColl(String corename) throws Exception {
                 + " collection: "
                 + Utils.toJSONString(rsp.getValues()));
       }
-      TimeOut timeOut = new TimeOut(3, TimeUnit.SECONDS, TimeSource.NANO_TIME);
-      for (; ; ) {
-        if (cores.getZkController().getClusterState().getCollectionOrNull(SYSTEM_COLL) != null) {
-          break;
-        } else {
-          if (timeOut.hasTimedOut()) {
-            throw new SolrException(
-                ErrorCode.SERVER_ERROR,
-                "Could not find " + SYSTEM_COLL + " collection even after 3 seconds");
-          }
-          timeOut.sleep(50);
-        }
+
+      try {
+        cores
+            .getZkController()
+            .getZkStateReader()
+            .waitForState(SYSTEM_COLL, 3, TimeUnit.SECONDS, Objects::nonNull);
+      } catch (TimeoutException e) {
+        throw new SolrException(
+            ErrorCode.SERVER_ERROR,
+            "Could not find " + SYSTEM_COLL + " collection even after 3 seconds");
       }
 
       action = RETRY;
diff --git a/solr/core/src/test/org/apache/solr/cloud/ChaosMonkeyNothingIsSafeWithPullReplicasTest.java b/solr/core/src/test/org/apache/solr/cloud/ChaosMonkeyNothingIsSafeWithPullReplicasTest.java
index ddb72aad118..6652009774f 100644
--- a/solr/core/src/test/org/apache/solr/cloud/ChaosMonkeyNothingIsSafeWithPullReplicasTest.java
+++ b/solr/core/src/test/org/apache/solr/cloud/ChaosMonkeyNothingIsSafeWithPullReplicasTest.java
@@ -343,7 +343,7 @@ public void test() throws Exception {
       List<Integer> numShardsNumReplicas = new ArrayList<>(2);
       numShardsNumReplicas.add(1);
       numShardsNumReplicas.add(1 + getPullReplicaCount());
-      checkForCollection("testcollection", numShardsNumReplicas, null);
+      checkForCollection("testcollection", numShardsNumReplicas);
 
       testSuccessful = true;
     } finally {
diff --git a/solr/core/src/test/org/apache/solr/cloud/ChaosMonkeySafeLeaderWithPullReplicasTest.java b/solr/core/src/test/org/apache/solr/cloud/ChaosMonkeySafeLeaderWithPullReplicasTest.java
index 764035dc043..b097ebea968 100644
--- a/solr/core/src/test/org/apache/solr/cloud/ChaosMonkeySafeLeaderWithPullReplicasTest.java
+++ b/solr/core/src/test/org/apache/solr/cloud/ChaosMonkeySafeLeaderWithPullReplicasTest.java
@@ -255,7 +255,7 @@ public void test() throws Exception {
     List<Integer> numShardsNumReplicas = new ArrayList<>(2);
     numShardsNumReplicas.add(1);
     numShardsNumReplicas.add(1 + getPullReplicaCount());
-    checkForCollection("testcollection", numShardsNumReplicas, null);
+    checkForCollection("testcollection", numShardsNumReplicas);
   }
 
   private void tryDelete() throws Exception {
diff --git a/solr/core/src/test/org/apache/solr/cloud/DeleteReplicaTest.java b/solr/core/src/test/org/apache/solr/cloud/DeleteReplicaTest.java
index 686a0a2554a..16242bcc5eb 100644
--- a/solr/core/src/test/org/apache/solr/cloud/DeleteReplicaTest.java
+++ b/solr/core/src/test/org/apache/solr/cloud/DeleteReplicaTest.java
@@ -488,11 +488,13 @@ private JettySolrRunner getJettyForReplica(Replica replica) {
   }
 
   private void waitForNodeLeave(String lostNodeName) throws InterruptedException {
-    ZkStateReader reader = cluster.getZkStateReader();
-    TimeOut timeOut = new TimeOut(20, TimeUnit.SECONDS, TimeSource.NANO_TIME);
-    while (reader.getClusterState().getLiveNodes().contains(lostNodeName)) {
-      Thread.sleep(100);
-      if (timeOut.hasTimedOut()) fail("Wait for " + lostNodeName + " to leave failed!");
+
+    try {
+      cluster
+          .getZkStateReader()
+          .waitForLiveNodes(20, TimeUnit.SECONDS, (o, n) -> !n.contains(lostNodeName));
+    } catch (TimeoutException e) {
+      fail("Wait for " + lostNodeName + " to leave failed!");
     }
   }
 
diff --git a/solr/core/src/test/org/apache/solr/cloud/HttpPartitionTest.java b/solr/core/src/test/org/apache/solr/cloud/HttpPartitionTest.java
index 2f3f62ad763..0d4f1c2b722 100644
--- a/solr/core/src/test/org/apache/solr/cloud/HttpPartitionTest.java
+++ b/solr/core/src/test/org/apache/solr/cloud/HttpPartitionTest.java
@@ -30,6 +30,7 @@
 import java.util.Map;
 import java.util.Set;
 import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
 import org.apache.lucene.tests.util.LuceneTestCase;
 import org.apache.solr.JSONTestUtil;
 import org.apache.solr.SolrTestCaseJ4.SuppressSSL;
@@ -48,13 +49,11 @@
 import org.apache.solr.common.cloud.ZkCoreNodeProps;
 import org.apache.solr.common.cloud.ZkStateReader;
 import org.apache.solr.common.util.NamedList;
-import org.apache.solr.common.util.TimeSource;
 import org.apache.solr.core.CoreContainer;
 import org.apache.solr.core.SolrCore;
 import org.apache.solr.embedded.JettySolrRunner;
 import org.apache.solr.util.RTimer;
 import org.apache.solr.util.TestInjection;
-import org.apache.solr.util.TimeOut;
 import org.apache.zookeeper.KeeperException;
 import org.junit.BeforeClass;
 import org.junit.Test;
@@ -291,27 +290,23 @@ protected void testRf2() throws Exception {
 
   protected void waitForState(String collection, String replicaName, Replica.State state, long ms)
       throws KeeperException, InterruptedException {
-    TimeOut timeOut = new TimeOut(ms, TimeUnit.MILLISECONDS, TimeSource.NANO_TIME);
-    Replica.State replicaState = Replica.State.ACTIVE;
-    while (!timeOut.hasTimedOut()) {
-      ZkStateReader zkr = ZkStateReader.from(cloudClient);
-      zkr.forceUpdateCollection(collection); // force the state to be fresh
-      ClusterState cs = zkr.getClusterState();
-      Collection<Slice> slices = cs.getCollection(collection).getActiveSlices();
-      Slice slice = slices.iterator().next();
-      Replica partitionedReplica = slice.getReplica(replicaName);
-      replicaState = partitionedReplica.getState();
-      if (replicaState == state) return;
+    ZkStateReader zkr = ZkStateReader.from(cloudClient);
+
+    try {
+      zkr.waitForState(
+          collection,
+          ms,
+          TimeUnit.MILLISECONDS,
+          c -> {
+            Collection<Slice> slices = c.getActiveSlices();
+            Slice slice = slices.iterator().next();
+            Replica partitionedReplica = slice.getReplica(replicaName);
+            Replica.State replicaState = partitionedReplica.getState();
+            return replicaState == state;
+          });
+    } catch (TimeoutException e) {
+      fail("Timeout waiting for state " + state + " of replica " + replicaName);
     }
-    assertEquals(
-        "Timeout waiting for state "
-            + state
-            + " of replica "
-            + replicaName
-            + ", current state "
-            + replicaState,
-        state,
-        replicaState);
   }
 
   protected void testRf3() throws Exception {
diff --git a/solr/core/src/test/org/apache/solr/cloud/LeaderFailureAfterFreshStartTest.java b/solr/core/src/test/org/apache/solr/cloud/LeaderFailureAfterFreshStartTest.java
index 5b7ff3f0428..1bc87e4f4e1 100644
--- a/solr/core/src/test/org/apache/solr/cloud/LeaderFailureAfterFreshStartTest.java
+++ b/solr/core/src/test/org/apache/solr/cloud/LeaderFailureAfterFreshStartTest.java
@@ -39,8 +39,6 @@
 import org.apache.solr.common.cloud.Replica;
 import org.apache.solr.common.cloud.ZkStateReader;
 import org.apache.solr.common.params.ModifiableSolrParams;
-import org.apache.solr.common.util.TimeSource;
-import org.apache.solr.util.TimeOut;
 import org.junit.Test;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
@@ -154,11 +152,7 @@ public void test() throws Exception {
       // shutdown the original leader
       log.info("Now shutting down initial leader");
       forceNodeFailures(singletonList(initialLeaderJetty));
-      waitForNewLeader(
-          cloudClient,
-          "shard1",
-          (Replica) initialLeaderJetty.client.info,
-          new TimeOut(15, TimeUnit.SECONDS, TimeSource.NANO_TIME));
+      waitForNewLeader(cloudClient, "shard1", (Replica) initialLeaderJetty.client.info);
       waitTillNodesActive();
       log.info("Updating mappings from zk");
       updateMappingsFromZk(jettys, clients, true);
diff --git a/solr/core/src/test/org/apache/solr/cloud/PeerSyncReplicationTest.java b/solr/core/src/test/org/apache/solr/cloud/PeerSyncReplicationTest.java
index 7042c3018e2..d9e501e69b6 100644
--- a/solr/core/src/test/org/apache/solr/cloud/PeerSyncReplicationTest.java
+++ b/solr/core/src/test/org/apache/solr/cloud/PeerSyncReplicationTest.java
@@ -44,10 +44,8 @@
 import org.apache.solr.common.cloud.Replica;
 import org.apache.solr.common.cloud.ZkStateReader;
 import org.apache.solr.common.params.ModifiableSolrParams;
-import org.apache.solr.common.util.TimeSource;
 import org.apache.solr.core.CoreContainer;
 import org.apache.solr.metrics.SolrMetricManager;
-import org.apache.solr.util.TimeOut;
 import org.junit.Test;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
@@ -154,11 +152,7 @@ public void test() throws Exception {
       log.info("Now shutting down initial leader");
       forceNodeFailures(singletonList(initialLeaderJetty));
       log.info("Updating mappings from zk");
-      waitForNewLeader(
-          cloudClient,
-          "shard1",
-          (Replica) initialLeaderJetty.client.info,
-          new TimeOut(15, TimeUnit.SECONDS, TimeSource.NANO_TIME));
+      waitForNewLeader(cloudClient, "shard1", (Replica) initialLeaderJetty.client.info);
       updateMappingsFromZk(jettys, clients, true);
       assertEquals(
           "PeerSynced node did not become leader",
diff --git a/solr/core/src/test/org/apache/solr/cloud/TestPullReplica.java b/solr/core/src/test/org/apache/solr/cloud/TestPullReplica.java
index f5cf82c701e..ec0564c249d 100644
--- a/solr/core/src/test/org/apache/solr/cloud/TestPullReplica.java
+++ b/solr/core/src/test/org/apache/solr/cloud/TestPullReplica.java
@@ -749,20 +749,13 @@ static void waitForNumDocsInAllReplicas(
     }
   }
 
-  static void waitForDeletion(String collection) throws InterruptedException, KeeperException {
-    TimeOut t = new TimeOut(10, TimeUnit.SECONDS, TimeSource.NANO_TIME);
-    while (cluster.getSolrClient().getClusterState().hasCollection(collection)) {
-      log.info("Collection not yet deleted");
-      try {
-        Thread.sleep(100);
-        if (t.hasTimedOut()) {
-          fail("Timed out waiting for collection " + collection + " to be deleted.");
-        }
-        cluster.getZkStateReader().forceUpdateCollection(collection);
-      } catch (SolrException e) {
-        return;
-      }
-    }
+  static void waitForDeletion(String collection) {
+    waitForState(
+        "Waiting for collection " + collection + " to be deleted",
+        collection,
+        (n, c) -> c == null,
+        10,
+        TimeUnit.SECONDS);
   }
 
   private DocCollection assertNumberOfReplicas(
diff --git a/solr/core/src/test/org/apache/solr/cloud/TestPullReplicaErrorHandling.java b/solr/core/src/test/org/apache/solr/cloud/TestPullReplicaErrorHandling.java
index f4af183434c..0d68f2f1993 100644
--- a/solr/core/src/test/org/apache/solr/cloud/TestPullReplicaErrorHandling.java
+++ b/solr/core/src/test/org/apache/solr/cloud/TestPullReplicaErrorHandling.java
@@ -120,7 +120,7 @@ public void tearDown() throws Exception {
       log.info("tearDown deleting collection");
       CollectionAdminRequest.deleteCollection(collectionName).process(cluster.getSolrClient());
       log.info("Collection deleted");
-      waitForDeletion(collectionName);
+      TestPullReplica.waitForDeletion(collectionName);
     }
     collectionName = null;
     super.tearDown();
@@ -347,22 +347,6 @@ protected SocketProxy getProxyForReplica(Replica replica) throws Exception {
     return proxy;
   }
 
-  private void waitForDeletion(String collection) throws InterruptedException, KeeperException {
-    TimeOut t = new TimeOut(10, TimeUnit.SECONDS, TimeSource.NANO_TIME);
-    while (cluster.getSolrClient().getClusterState().hasCollection(collection)) {
-      log.info("Collection not yet deleted");
-      try {
-        Thread.sleep(100);
-        if (t.hasTimedOut()) {
-          fail("Timed out waiting for collection " + collection + " to be deleted.");
-        }
-        cluster.getZkStateReader().forceUpdateCollection(collection);
-      } catch (SolrException e) {
-        return;
-      }
-    }
-  }
-
   private CollectionStatePredicate activeReplicaCount(
       int numWriter, int numActive, int numPassive) {
     return (liveNodes, collectionState) -> {
diff --git a/solr/core/src/test/org/apache/solr/cloud/TestRebalanceLeaders.java b/solr/core/src/test/org/apache/solr/cloud/TestRebalanceLeaders.java
index 0b59d5d74ab..20343487523 100644
--- a/solr/core/src/test/org/apache/solr/cloud/TestRebalanceLeaders.java
+++ b/solr/core/src/test/org/apache/solr/cloud/TestRebalanceLeaders.java
@@ -111,8 +111,7 @@ public void removeAllProperties() {
   // shardUnique=true for the special property preferredLeader. That was removed at one point, so
   // we're explicitly testing that as well.
   @Test
-  public void testSetArbitraryPropertySliceUnique()
-      throws IOException, SolrServerException, InterruptedException {
+  public void testSetArbitraryPropertySliceUnique() throws IOException, SolrServerException {
     // Check both special (preferredLeader) and something arbitrary.
     doTestSetArbitraryPropertySliceUnique("foo" + random().nextInt(1_000_000));
     removeAllProperties();
@@ -124,8 +123,7 @@ public void testSetArbitraryPropertySliceUnique()
   // individual properties on individual nodes. This one relies on Solr to pick which replicas to
   // set the property on
   @Test
-  public void testBalancePropertySliceUnique()
-      throws InterruptedException, IOException, SolrServerException {
+  public void testBalancePropertySliceUnique() throws IOException, SolrServerException {
     // Check both cases of "special" property preferred(Ll)eader
     doTestBalancePropertySliceUnique("foo" + random().nextInt(1_000_000));
     removeAllProperties();
@@ -160,7 +158,7 @@ public void testRebalanceLeaders() throws Exception {
   // on an individual
   // replica.
   private void doTestSetArbitraryPropertySliceUnique(String propIn)
-      throws InterruptedException, IOException, SolrServerException {
+      throws IOException, SolrServerException {
     final String prop = (random().nextBoolean()) ? propIn : propIn.toUpperCase(Locale.ROOT);
     // First set the property in some replica in some slice
     forceUpdateCollectionStatus();
@@ -176,43 +174,29 @@ private void doTestSetArbitraryPropertySliceUnique(String propIn)
       Replica rep = reps[random().nextInt(reps.length)];
       // Set the property on a particular replica
       setProp(slice, rep, prop);
-      TimeOut timeout = new TimeOut(timeoutMs, TimeUnit.MILLISECONDS, TimeSource.NANO_TIME);
-
-      long count = 0;
-      boolean rightRep = false;
-      Slice modSlice;
-      DocCollection modColl = null; // keeps IDE happy
 
       // ensure that no other replica in that slice has the property when we return.
-      while (timeout.hasTimedOut() == false) {
-        forceUpdateCollectionStatus();
-        modColl = cluster.getSolrClient().getClusterState().getCollection(COLLECTION_NAME);
-        modSlice = modColl.getSlice(slice.getName());
-        rightRep =
-            modSlice
-                .getReplica(rep.getName())
-                .getBool("property." + prop.toLowerCase(Locale.ROOT), false);
-        count =
-            modSlice.getReplicas().stream()
-                .filter(
-                    thisRep -> thisRep.getBool("property." + prop.toLowerCase(Locale.ROOT), false))
-                .count();
-
-        if (count == 1 && rightRep) {
-          break;
-        }
-
-        TimeUnit.MILLISECONDS.sleep(50);
-      }
-      if (count != 1 || rightRep == false) {
-        fail(
-            "The property "
-                + prop
-                + " was not uniquely distributed in slice "
-                + slice.getName()
-                + " "
-                + modColl.toString());
-      }
+      waitForState(
+          "Check property is uniquely distributed in slice: " + prop,
+          COLLECTION_NAME,
+          (n, c) -> {
+            forceUpdateCollectionStatus();
+            Slice modSlice = c.getSlice(slice.getName());
+            boolean rightRep =
+                modSlice
+                    .getReplica(rep.getName())
+                    .getBool("property." + prop.toLowerCase(Locale.ROOT), false);
+            long count =
+                modSlice.getReplicas().stream()
+                    .filter(
+                        thisRep ->
+                            thisRep.getBool("property." + prop.toLowerCase(Locale.ROOT), false))
+                    .count();
+
+            return count == 1 && rightRep;
+          },
+          timeoutMs,
+          TimeUnit.MILLISECONDS);
     }
   }
 
@@ -332,7 +316,7 @@ private void sendRebalanceCommand() throws SolrServerException, IOException {
   // re-assigned with the
   // BALANCESHARDUNIQUE command.
   private void doTestBalancePropertySliceUnique(String propIn)
-      throws InterruptedException, IOException, SolrServerException {
+      throws IOException, SolrServerException {
     final String prop = (random().nextBoolean()) ? propIn : propIn.toUpperCase(Locale.ROOT);
 
     // Concentrate the properties on as few replicas a possible
@@ -350,63 +334,55 @@ private void doTestBalancePropertySliceUnique(String propIn)
 
   private void verifyPropCorrectlyDistributed(String prop) {
 
-    TimeOut timeout = new TimeOut(timeoutMs, TimeUnit.MILLISECONDS, TimeSource.NANO_TIME);
-
     String propLC = prop.toLowerCase(Locale.ROOT);
-    DocCollection docCollection = null;
-    while (timeout.hasTimedOut() == false) {
-      forceUpdateCollectionStatus();
-      docCollection = cluster.getSolrClient().getClusterState().getCollection(COLLECTION_NAME);
-      int maxPropCount = Integer.MAX_VALUE;
-      int minPropCount = Integer.MIN_VALUE;
-      for (Slice slice : docCollection.getSlices()) {
-        int repCount = 0;
-        for (Replica rep : slice.getReplicas()) {
-          if (rep.getBool("property." + propLC, false)) {
-            repCount++;
+    waitForState(
+        "Check property is distributed evenly: " + prop,
+        COLLECTION_NAME,
+        (liveNodes, docCollection) -> {
+          int maxPropCount = 0;
+          int minPropCount = Integer.MAX_VALUE;
+          for (Slice slice : docCollection.getSlices()) {
+            int repCount = 0;
+            for (Replica rep : slice.getReplicas()) {
+              if (rep.getBool("property." + propLC, false)) {
+                repCount++;
+              }
+            }
+            maxPropCount = Math.max(maxPropCount, repCount);
+            minPropCount = Math.min(minPropCount, repCount);
           }
-        }
-        maxPropCount = Math.max(maxPropCount, repCount);
-        minPropCount = Math.min(minPropCount, repCount);
-      }
-      if (Math.abs(maxPropCount - minPropCount) < 2) return;
-    }
-    log.error("Property {} is not distributed evenly. {}", prop, docCollection);
-    fail("Property is not distributed evenly " + prop);
+          return Math.abs(maxPropCount - minPropCount) < 2;
+        },
+        timeoutMs,
+        TimeUnit.MILLISECONDS);
   }
 
   // Used when we concentrate the leader on a few nodes.
   private void verifyPropDistributedAsExpected(
-      Map<String, String> expectedShardReplicaMap, String prop) throws InterruptedException {
+      Map<String, String> expectedShardReplicaMap, String prop) {
     // Make sure that the shard unique are where you expect.
-    TimeOut timeout = new TimeOut(timeoutMs, TimeUnit.MILLISECONDS, TimeSource.NANO_TIME);
-
     String propLC = prop.toLowerCase(Locale.ROOT);
-    boolean failure = false;
-    DocCollection docCollection = null;
-    while (timeout.hasTimedOut() == false) {
-      forceUpdateCollectionStatus();
-      docCollection = cluster.getSolrClient().getClusterState().getCollection(COLLECTION_NAME);
-      failure = false;
-      for (Map.Entry<String, String> ent : expectedShardReplicaMap.entrySet()) {
-        Replica rep = docCollection.getSlice(ent.getKey()).getReplica(ent.getValue());
-        if (rep.getBool("property." + propLC, false) == false) {
-          failure = true;
-        }
-      }
-      if (failure == false) {
-        return;
-      }
-      TimeUnit.MILLISECONDS.sleep(100);
-    }
+    String message =
+        String.format(
+            Locale.ROOT,
+            "Checking properties are on the expected replicas. Props:%s Expected:%s",
+            prop,
+            expectedShardReplicaMap.toString());
 
-    fail(
-        prop
-            + " properties are not on the expected replicas: "
-            + docCollection.toString()
-            + System.lineSeparator()
-            + "Expected "
-            + expectedShardReplicaMap.toString());
+    waitForState(
+        message,
+        COLLECTION_NAME,
+        (liveNodes, docCollection) -> {
+          for (Map.Entry<String, String> ent : expectedShardReplicaMap.entrySet()) {
+            Replica rep = docCollection.getSlice(ent.getKey()).getReplica(ent.getValue());
+            if (rep.getBool("property." + propLC, false) == false) {
+              return false;
+            }
+          }
+          return true;
+        },
+        timeoutMs,
+        TimeUnit.MILLISECONDS);
   }
 
   // Just check that the property is distributed as expectecd. This does _not_ rebalance the leaders
@@ -558,14 +534,6 @@ private void concentratePropByRestartingJettys() throws Exception {
     // then start them again to concentrate the leaders. It's not necessary that all shards have a
     // leader.
 
-    ExecutorService executorService = ExecutorUtil.newMDCAwareCachedThreadPool("Start Jetty");
-
-    for (JettySolrRunner jetty : jettys) {
-      cluster.stopJettySolrRunner(jetty);
-    }
-
-    ExecutorUtil.shutdownAndAwaitTermination(executorService);
-
     for (JettySolrRunner jetty : jettys) {
       cluster.stopJettySolrRunner(jetty);
     }
@@ -575,7 +543,7 @@ private void concentratePropByRestartingJettys() throws Exception {
     }
     checkReplicasInactive(jettys);
 
-    executorService = ExecutorUtil.newMDCAwareCachedThreadPool("Start Jetty");
+    ExecutorService executorService = ExecutorUtil.newMDCAwareCachedThreadPool("Start Jetty");
 
     for (int idx = 0; idx < jettys.size(); ++idx) {
       int finalIdx = idx;
@@ -604,74 +572,61 @@ private void forceUpdateCollectionStatus() {
 
   // Since we have to restart jettys, we don't want to try re-balancing etc. until we're sure all
   // jettys that should be up are and all replicas are active.
-  private void checkReplicasInactive(List<JettySolrRunner> downJettys) throws InterruptedException {
-    TimeOut timeout = new TimeOut(timeoutMs, TimeUnit.MILLISECONDS, TimeSource.NANO_TIME);
-    DocCollection docCollection = null;
-    Set<String> liveNodes = null;
+  private void checkReplicasInactive(List<JettySolrRunner> downJettys) {
 
     Set<String> downJettyNodes = new TreeSet<>();
     for (JettySolrRunner jetty : downJettys) {
       downJettyNodes.add(
           jetty.getBaseUrl().getHost() + ":" + jetty.getBaseUrl().getPort() + "_solr");
     }
-    while (timeout.hasTimedOut() == false) {
-      forceUpdateCollectionStatus();
-      docCollection = cluster.getSolrClient().getClusterState().getCollection(COLLECTION_NAME);
-      liveNodes = cluster.getSolrClient().getClusterState().getLiveNodes();
-      boolean expectedInactive = true;
-
-      for (Slice slice : docCollection.getSlices()) {
-        for (Replica rep : slice.getReplicas()) {
-          if (downJettyNodes.contains(rep.getNodeName()) == false) {
-            continue; // We are on a live node
-          }
-          // A replica on an allegedly down node is reported as active.
-          if (rep.isActive(liveNodes)) {
-            expectedInactive = false;
+
+    waitForState(
+        "Waiting for all replicas to become inactive",
+        COLLECTION_NAME,
+        (liveNodes, docCollection) -> {
+          boolean expectedInactive = true;
+
+          for (Slice slice : docCollection.getSlices()) {
+            for (Replica rep : slice.getReplicas()) {
+              if (downJettyNodes.contains(rep.getNodeName()) == false) {
+                continue; // We are on a live node
+              }
+              // A replica on an allegedly down node is reported as active.
+              if (rep.isActive(liveNodes)) {
+                expectedInactive = false;
+              }
+            }
           }
-        }
-      }
-      if (expectedInactive) {
-        return;
-      }
-      TimeUnit.MILLISECONDS.sleep(100);
-    }
-    fail(
-        "timed out waiting for all replicas to become inactive: livenodes: "
-            + liveNodes
-            + " Collection state: "
-            + docCollection.toString());
+          return expectedInactive;
+        },
+        timeoutMs,
+        TimeUnit.MILLISECONDS);
   }
 
   // We need to wait around until all replicas are active before expecting rebalancing or
   // distributing shard-unique properties to work.
-  private void checkAllReplicasActive() throws InterruptedException {
-    TimeOut timeout = new TimeOut(timeoutMs, TimeUnit.MILLISECONDS, TimeSource.NANO_TIME);
-    while (timeout.hasTimedOut() == false) {
-      forceUpdateCollectionStatus();
-      DocCollection docCollection =
-          cluster.getSolrClient().getClusterState().getCollection(COLLECTION_NAME);
-      Set<String> liveNodes = cluster.getSolrClient().getClusterState().getLiveNodes();
-      boolean allActive = true;
-      for (Slice slice : docCollection.getSlices()) {
-        for (Replica rep : slice.getReplicas()) {
-          if (rep.isActive(liveNodes) == false) {
-            allActive = false;
+  private void checkAllReplicasActive() {
+    waitForState(
+        "Waiting for all replicas to become active",
+        COLLECTION_NAME,
+        (liveNodes, docCollection) -> {
+          boolean allActive = true;
+          for (Slice slice : docCollection.getSlices()) {
+            for (Replica rep : slice.getReplicas()) {
+              if (rep.isActive(liveNodes) == false) {
+                allActive = false;
+              }
+            }
           }
-        }
-      }
-      if (allActive) {
-        return;
-      }
-      TimeUnit.MILLISECONDS.sleep(100);
-    }
-    fail("timed out waiting for all replicas to become active");
+          return allActive;
+        },
+        timeoutMs,
+        TimeUnit.MILLISECONDS);
   }
 
   // use a simple heuristic to put as many replicas with the property on as few nodes as possible.
   // The point is that then we can execute BALANCESHARDUNIQUE and be sure it worked correctly
-  private void concentrateProp(String prop)
-      throws InterruptedException, IOException, SolrServerException {
+  private void concentrateProp(String prop) throws IOException, SolrServerException {
     // find all the live nodes for each slice, assign the leader to the first replica that is in the
     // lowest position on live_nodes
     List<String> liveNodes =
@@ -704,43 +659,26 @@ private void concentrateProp(String prop)
   }
 
   // make sure that the property in question is unique per shard.
-  private Map<String, String> verifyPropUniquePerShard(String prop) throws InterruptedException {
-    Map<String, String> uniquePropMaps = new TreeMap<>();
-
-    TimeOut timeout = new TimeOut(timeoutMs, TimeUnit.MILLISECONDS, TimeSource.NANO_TIME);
-    while (timeout.hasTimedOut() == false) {
-      uniquePropMaps.clear();
-      if (checkUniquePropPerShard(uniquePropMaps, prop)) {
-        return uniquePropMaps;
-      }
-      TimeUnit.MILLISECONDS.sleep(10);
-    }
-    fail(
-        "There should be exactly one replica with value "
-            + prop
-            + " set to true per shard: "
-            + cluster.getSolrClient().getClusterState().getCollection(COLLECTION_NAME).toString());
-    return null; // keeps IDE happy.
-  }
-
-  // return true if every shard has exactly one replica with the unique property set to "true"
-  private boolean checkUniquePropPerShard(Map<String, String> uniques, String prop) {
-    forceUpdateCollectionStatus();
-    DocCollection docCollection =
-        cluster.getSolrClient().getClusterState().getCollection(COLLECTION_NAME);
+  private void verifyPropUniquePerShard(String prop) {
 
-    for (Slice slice : docCollection.getSlices()) {
-      int propCount = 0;
-      for (Replica rep : slice.getReplicas()) {
-        if (rep.getBool("property." + prop.toLowerCase(Locale.ROOT), false)) {
-          propCount++;
-          uniques.put(slice.getName(), rep.getName());
-        }
-      }
-      if (1 != propCount) {
-        return false;
-      }
-    }
-    return true;
+    waitForState(
+        "Waiting to have exactly one replica with " + prop + "set per shard",
+        COLLECTION_NAME,
+        (liveNodes, docCollection) -> {
+          for (Slice slice : docCollection.getSlices()) {
+            int propCount = 0;
+            for (Replica rep : slice.getReplicas()) {
+              if (rep.getBool("property." + prop.toLowerCase(Locale.ROOT), false)) {
+                propCount++;
+              }
+            }
+            if (1 != propCount) {
+              return false;
+            }
+          }
+          return true;
+        },
+        timeoutMs,
+        TimeUnit.MILLISECONDS);
   }
 }
diff --git a/solr/core/src/test/org/apache/solr/cloud/TestTlogReplica.java b/solr/core/src/test/org/apache/solr/cloud/TestTlogReplica.java
index 563ed686298..d559de6d333 100644
--- a/solr/core/src/test/org/apache/solr/cloud/TestTlogReplica.java
+++ b/solr/core/src/test/org/apache/solr/cloud/TestTlogReplica.java
@@ -815,17 +815,17 @@ public void testRebalanceLeaders() throws Exception {
     cloudClient.request(request);
 
     // Wait until a preferredleader flag is set to the new leader candidate
-    TimeOut timeout = new TimeOut(10, TimeUnit.SECONDS, TimeSource.NANO_TIME);
-    while (!timeout.hasTimedOut()) {
-      Map<String, Slice> slices =
-          cloudClient.getClusterState().getCollection(collectionName).getSlicesMap();
-      Replica me = slices.get(slice.getName()).getReplica(newLeader.getName());
-      if (me.getBool("property.preferredleader", false)) {
-        break;
-      }
-      Thread.sleep(100);
-    }
-    assertFalse("Timeout waiting for setting preferredleader flag", timeout.hasTimedOut());
+    String newLeaderName = newLeader.getName();
+    waitForState(
+        "Waiting for setting preferredleader flag",
+        collectionName,
+        (n, c) -> {
+          Map<String, Slice> slices = c.getSlicesMap();
+          Replica me = slices.get(slice.getName()).getReplica(newLeaderName);
+          return me.getBool("property.preferredleader", false);
+        },
+        10,
+        TimeUnit.SECONDS);
 
     // Rebalance leaders
     params = new ModifiableSolrParams();
@@ -837,18 +837,17 @@ public void testRebalanceLeaders() throws Exception {
     cloudClient.request(request);
 
     // Wait until a new leader is elected
-    timeout = new TimeOut(30, TimeUnit.SECONDS, TimeSource.NANO_TIME);
-    while (!timeout.hasTimedOut()) {
-      docCollection = getCollectionState(collectionName);
-      Replica leader = docCollection.getSlice(slice.getName()).getLeader();
-      if (leader != null
-          && leader.getName().equals(newLeader.getName())
-          && leader.isActive(cloudClient.getClusterState().getLiveNodes())) {
-        break;
-      }
-      Thread.sleep(100);
-    }
-    assertFalse("Timeout waiting for a new leader to be elected", timeout.hasTimedOut());
+    waitForState(
+        "Waiting for a new leader to be elected",
+        collectionName,
+        (n, c) -> {
+          Replica leader = c.getSlice(slice.getName()).getLeader();
+          return leader != null
+              && leader.getName().equals(newLeaderName)
+              && leader.isActive(cloudClient.getClusterState().getLiveNodes());
+        },
+        30,
+        TimeUnit.SECONDS);
 
     new UpdateRequest()
         .add(sdoc("id", "1"))
@@ -1027,19 +1026,13 @@ private void waitForNumDocsInAllReplicas(
     }
   }
 
-  private void waitForDeletion(String collection) throws InterruptedException, KeeperException {
-    TimeOut t = new TimeOut(10, TimeUnit.SECONDS, TimeSource.NANO_TIME);
-    while (cluster.getSolrClient().getClusterState().hasCollection(collection)) {
-      try {
-        Thread.sleep(100);
-        if (t.hasTimedOut()) {
-          fail("Timed out waiting for collection " + collection + " to be deleted.");
-        }
-        cluster.getZkStateReader().forceUpdateCollection(collection);
-      } catch (SolrException e) {
-        return;
-      }
-    }
+  private void waitForDeletion(String collection) {
+    waitForState(
+        "Waiting for collection " + collection + " to be deleted",
+        collection,
+        (n, c) -> c == null,
+        10,
+        TimeUnit.SECONDS);
   }
 
   private DocCollection assertNumberOfReplicas(
diff --git a/solr/core/src/test/org/apache/solr/cloud/api/collections/ShardSplitTest.java b/solr/core/src/test/org/apache/solr/cloud/api/collections/ShardSplitTest.java
index 43f3367f489..28335c920eb 100644
--- a/solr/core/src/test/org/apache/solr/cloud/api/collections/ShardSplitTest.java
+++ b/solr/core/src/test/org/apache/solr/cloud/api/collections/ShardSplitTest.java
@@ -960,7 +960,7 @@ public void splitByRouteFieldTest() throws Exception {
     }
 
     List<Integer> list = collectionInfos.get(collectionName);
-    checkForCollection(collectionName, list, null);
+    checkForCollection(collectionName, list);
 
     waitForRecoveriesToFinish(false);
 
@@ -1031,7 +1031,7 @@ private void splitByRouteKeyTest() throws Exception {
     }
 
     List<Integer> list = collectionInfos.get(collectionName);
-    checkForCollection(collectionName, list, null);
+    checkForCollection(collectionName, list);
 
     waitForRecoveriesToFinish(false);
 
diff --git a/solr/modules/hdfs/src/test/org/apache/solr/hdfs/cloud/SharedFileSystemAutoReplicaFailoverTest.java b/solr/modules/hdfs/src/test/org/apache/solr/hdfs/cloud/SharedFileSystemAutoReplicaFailoverTest.java
index 329cc7fee11..34e3b2a1492 100644
--- a/solr/modules/hdfs/src/test/org/apache/solr/hdfs/cloud/SharedFileSystemAutoReplicaFailoverTest.java
+++ b/solr/modules/hdfs/src/test/org/apache/solr/hdfs/cloud/SharedFileSystemAutoReplicaFailoverTest.java
@@ -31,6 +31,7 @@
 import java.util.concurrent.Future;
 import java.util.concurrent.SynchronousQueue;
 import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
 import java.util.stream.Collectors;
 import org.apache.hadoop.hdfs.MiniDFSCluster;
 import org.apache.lucene.tests.util.LuceneTestCase;
@@ -57,13 +58,11 @@
 import org.apache.solr.common.params.SolrParams;
 import org.apache.solr.common.util.ExecutorUtil;
 import org.apache.solr.common.util.SolrNamedThreadFactory;
-import org.apache.solr.common.util.TimeSource;
 import org.apache.solr.core.CoreDescriptor;
 import org.apache.solr.embedded.JettySolrRunner;
 import org.apache.solr.hdfs.util.BadHdfsThreadsFilter;
 import org.apache.solr.util.LogLevel;
 import org.apache.solr.util.TestInjection;
-import org.apache.solr.util.TimeOut;
 import org.junit.AfterClass;
 import org.junit.Before;
 import org.junit.BeforeClass;
@@ -425,37 +424,43 @@ private boolean waitingForReplicasNotLive(
 
   private void assertSliceAndReplicaCount(
       String collection, int numSlices, int numReplicas, int timeOutInMs)
-      throws InterruptedException, IOException {
-    TimeOut timeOut = new TimeOut(timeOutInMs, TimeUnit.MILLISECONDS, TimeSource.NANO_TIME);
-    while (!timeOut.hasTimedOut()) {
-      ClusterState clusterState = cloudClient.getClusterState();
-      Collection<Slice> slices = clusterState.getCollection(collection).getActiveSlices();
-      if (slices.size() == numSlices) {
-        boolean isMatch = true;
-        for (Slice slice : slices) {
-          int count = 0;
-          for (Replica replica : slice.getReplicas()) {
-            if (replica.getState() == Replica.State.ACTIVE
-                && clusterState.liveNodesContain(replica.getNodeName())) {
-              count++;
-            }
-          }
-          if (count < numReplicas) {
-            isMatch = false;
-          }
-        }
-        if (isMatch) return;
-      }
-      Thread.sleep(200);
+      throws InterruptedException {
+
+    try {
+      ZkStateReader.from(cloudClient)
+          .waitForState(
+              collection,
+              timeOutInMs,
+              TimeUnit.MILLISECONDS,
+              (liveNodes, c) -> {
+                Collection<Slice> slices = c.getActiveSlices();
+                if (slices.size() == numSlices) {
+                  for (Slice slice : slices) {
+                    int count = 0;
+                    for (Replica replica : slice.getReplicas()) {
+                      if (replica.getState() == Replica.State.ACTIVE
+                          && liveNodes.contains(replica.getNodeName())) {
+                        count++;
+                      }
+                    }
+                    if (count < numReplicas) {
+                      return false;
+                    }
+                  }
+                  return true;
+                }
+                return false;
+              });
+    } catch (TimeoutException e) {
+      fail(
+          "Expected numSlices="
+              + numSlices
+              + " numReplicas="
+              + numReplicas
+              + " but found "
+              + cloudClient.getClusterState().getCollection(collection)
+              + " with /live_nodes: "
+              + cloudClient.getClusterState().getLiveNodes());
     }
-    fail(
-        "Expected numSlices="
-            + numSlices
-            + " numReplicas="
-            + numReplicas
-            + " but found "
-            + cloudClient.getClusterState().getCollection(collection)
-            + " with /live_nodes: "
-            + cloudClient.getClusterState().getLiveNodes());
   }
 }
diff --git a/solr/modules/hdfs/src/test/org/apache/solr/hdfs/cloud/StressHdfsTest.java b/solr/modules/hdfs/src/test/org/apache/solr/hdfs/cloud/StressHdfsTest.java
index 7bb98b8d97b..326daaddc03 100644
--- a/solr/modules/hdfs/src/test/org/apache/solr/hdfs/cloud/StressHdfsTest.java
+++ b/solr/modules/hdfs/src/test/org/apache/solr/hdfs/cloud/StressHdfsTest.java
@@ -48,9 +48,7 @@
 import org.apache.solr.common.params.CollectionParams.CollectionAction;
 import org.apache.solr.common.params.ModifiableSolrParams;
 import org.apache.solr.common.util.NamedList;
-import org.apache.solr.common.util.TimeSource;
 import org.apache.solr.hdfs.util.BadHdfsThreadsFilter;
-import org.apache.solr.util.TimeOut;
 import org.junit.AfterClass;
 import org.junit.BeforeClass;
 import org.junit.Test;
@@ -232,14 +230,8 @@ private void createAndDeleteCollection() throws Exception {
     request.setPath("/admin/collections");
     solrClient.request(request);
 
-    final TimeOut timeout = new TimeOut(10, TimeUnit.SECONDS, TimeSource.NANO_TIME);
-    while (cloudClient.getClusterState().hasCollection(DELETE_DATA_DIR_COLLECTION)) {
-      if (timeout.hasTimedOut()) {
-        throw new AssertionError("Timeout waiting to see removed collection leave clusterstate");
-      }
-
-      Thread.sleep(200);
-    }
+    waitForCollectionToDisappear(
+        DELETE_DATA_DIR_COLLECTION, ZkStateReader.from(cloudClient), true, 10);
 
     // check that all dirs are gone
     for (String dataDir : dataDirs) {
diff --git a/solr/test-framework/src/java/org/apache/solr/cloud/AbstractChaosMonkeyNothingIsSafeTestBase.java b/solr/test-framework/src/java/org/apache/solr/cloud/AbstractChaosMonkeyNothingIsSafeTestBase.java
index fb138876b30..8c8a115636e 100644
--- a/solr/test-framework/src/java/org/apache/solr/cloud/AbstractChaosMonkeyNothingIsSafeTestBase.java
+++ b/solr/test-framework/src/java/org/apache/solr/cloud/AbstractChaosMonkeyNothingIsSafeTestBase.java
@@ -303,7 +303,7 @@ public void test() throws Exception {
       List<Integer> numShardsNumReplicas = new ArrayList<>(2);
       numShardsNumReplicas.add(1);
       numShardsNumReplicas.add(1);
-      checkForCollection("testcollection", numShardsNumReplicas, null);
+      checkForCollection("testcollection", numShardsNumReplicas);
 
       testSuccessful = true;
     } finally {
diff --git a/solr/test-framework/src/java/org/apache/solr/cloud/AbstractChaosMonkeySafeLeaderTestBase.java b/solr/test-framework/src/java/org/apache/solr/cloud/AbstractChaosMonkeySafeLeaderTestBase.java
index ab765d32699..ef371efcad4 100644
--- a/solr/test-framework/src/java/org/apache/solr/cloud/AbstractChaosMonkeySafeLeaderTestBase.java
+++ b/solr/test-framework/src/java/org/apache/solr/cloud/AbstractChaosMonkeySafeLeaderTestBase.java
@@ -203,7 +203,7 @@ public void test() throws Exception {
     List<Integer> numShardsNumReplicas = new ArrayList<>(2);
     numShardsNumReplicas.add(1);
     numShardsNumReplicas.add(1);
-    checkForCollection("testcollection", numShardsNumReplicas, null);
+    checkForCollection("testcollection", numShardsNumReplicas);
   }
 
   private void tryDelete() throws Exception {
diff --git a/solr/test-framework/src/java/org/apache/solr/cloud/AbstractDistribZkTestBase.java b/solr/test-framework/src/java/org/apache/solr/cloud/AbstractDistribZkTestBase.java
index 37c90d98db9..f256a5fb937 100644
--- a/solr/test-framework/src/java/org/apache/solr/cloud/AbstractDistribZkTestBase.java
+++ b/solr/test-framework/src/java/org/apache/solr/cloud/AbstractDistribZkTestBase.java
@@ -16,7 +16,6 @@
  */
 package org.apache.solr.cloud;
 
-import static java.util.concurrent.TimeUnit.MILLISECONDS;
 import static java.util.concurrent.TimeUnit.SECONDS;
 
 import java.io.File;
@@ -41,7 +40,6 @@
 import org.apache.solr.core.Diagnostics;
 import org.apache.solr.core.MockDirectoryFactory;
 import org.apache.solr.embedded.JettySolrRunner;
-import org.apache.solr.util.TimeOut;
 import org.apache.zookeeper.KeeperException;
 import org.junit.BeforeClass;
 import org.slf4j.Logger;
@@ -242,65 +240,42 @@ public static void waitForCollectionToDisappear(
     log.info("Collection has disappeared - collection:{}", collection);
   }
 
-  static void waitForNewLeader(
-      CloudSolrClient cloudClient, String shardName, Replica oldLeader, TimeOut timeOut)
+  static void waitForNewLeader(CloudSolrClient cloudClient, String shardName, Replica oldLeader)
       throws Exception {
-    log.info("Will wait for a node to become leader for {} secs", timeOut.timeLeft(SECONDS));
+    log.info("Will wait for a node to become leader for 15 secs");
     ZkStateReader zkStateReader = ZkStateReader.from(cloudClient);
-    zkStateReader.forceUpdateCollection(DEFAULT_COLLECTION);
-
-    for (; ; ) {
-      ClusterState clusterState = zkStateReader.getClusterState();
-      DocCollection coll = clusterState.getCollection("collection1");
-      Slice slice = coll.getSlice(shardName);
-      if (slice.getLeader() != null
-          && !slice.getLeader().equals(oldLeader)
-          && slice.getLeader().getState() == Replica.State.ACTIVE) {
-        if (log.isInfoEnabled()) {
-          log.info(
-              "Old leader {}, new leader {}. New leader got elected in {} ms",
-              oldLeader,
-              slice.getLeader(),
-              timeOut.timeElapsed(MILLISECONDS));
-        }
-        break;
-      }
-
-      if (timeOut.hasTimedOut()) {
-        Diagnostics.logThreadDumps("Could not find new leader in specified timeout");
-        zkStateReader.getZkClient().printLayoutToStream(System.out);
-        fail(
-            "Could not find new leader even after waiting for "
-                + timeOut.timeElapsed(MILLISECONDS)
-                + "ms");
-      }
 
-      Thread.sleep(100);
-    }
+    long startNs = System.nanoTime();
+    try {
+      zkStateReader.waitForState(
+          "collection1",
+          15,
+          TimeUnit.SECONDS,
+          (docCollection) -> {
+            if (docCollection == null) return false;
 
-    zkStateReader.waitForState(
-        "collection1",
-        timeOut.timeLeft(SECONDS),
-        TimeUnit.SECONDS,
-        (docCollection) -> {
-          if (docCollection == null) return false;
-
-          Slice slice = docCollection.getSlice(shardName);
-          if (slice != null
-              && slice.getLeader() != null
-              && !slice.getLeader().equals(oldLeader)
-              && slice.getLeader().getState() == Replica.State.ACTIVE) {
-            if (log.isInfoEnabled()) {
-              log.info(
-                  "Old leader {}, new leader {}. New leader got elected in {} ms",
-                  oldLeader,
-                  slice.getLeader(),
-                  timeOut.timeElapsed(MILLISECONDS));
+            Slice slice = docCollection.getSlice(shardName);
+            if (slice != null
+                && slice.getLeader() != null
+                && !slice.getLeader().equals(oldLeader)
+                && slice.getLeader().getState() == Replica.State.ACTIVE) {
+              if (log.isInfoEnabled()) {
+                log.info(
+                    "Old leader {}, new leader {}. New leader got elected in {} ms",
+                    oldLeader,
+                    slice.getLeader(),
+                    TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - startNs));
+              }
+              return true;
             }
-            return true;
-          }
-          return false;
-        });
+            return false;
+          });
+    } catch (TimeoutException e) {
+      // If we failed to get a new leader, print some diagnotics before the test fails
+      Diagnostics.logThreadDumps("Could not find new leader in specified timeout");
+      zkStateReader.getZkClient().printLayoutToStream(System.out);
+      fail("Could not find new leader even after waiting for 15s");
+    }
   }
 
   public static void verifyReplicaStatus(
diff --git a/solr/test-framework/src/java/org/apache/solr/cloud/AbstractFullDistribZkTestBase.java b/solr/test-framework/src/java/org/apache/solr/cloud/AbstractFullDistribZkTestBase.java
index dcdab4e86d1..003ef951b0f 100644
--- a/solr/test-framework/src/java/org/apache/solr/cloud/AbstractFullDistribZkTestBase.java
+++ b/solr/test-framework/src/java/org/apache/solr/cloud/AbstractFullDistribZkTestBase.java
@@ -44,6 +44,7 @@
 import java.util.concurrent.TimeUnit;
 import java.util.concurrent.TimeoutException;
 import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicReference;
 import java.util.function.Consumer;
 import org.apache.solr.client.solrj.SolrClient;
 import org.apache.solr.client.solrj.SolrQuery;
@@ -2217,83 +2218,60 @@ public static SolrInputDocument getDoc(Object... fields) throws Exception {
     return sdoc(fields);
   }
 
-  private String checkCollectionExpectations(
-      String collectionName,
-      List<Integer> numShardsNumReplicaList,
-      List<String> nodesAllowedToRunShards)
-      throws IOException {
-    getCommonCloudSolrClient();
-    ClusterState clusterState = cloudClient.getClusterState();
-    int expectedSlices = numShardsNumReplicaList.get(0);
-    // The Math.min thing is here, because we expect replication-factor to be reduced to if there
-    // are not enough live nodes to spread all shards of a collection over different nodes
-    int expectedShardsPerSlice = numShardsNumReplicaList.get(1);
-    int expectedTotalShards = expectedSlices * expectedShardsPerSlice;
-
-    //      Map<String,DocCollection> collections = clusterState
-    //          .getCollectionStates();
-    if (clusterState.hasCollection(collectionName)) {
-      Map<String, Slice> slices = clusterState.getCollection(collectionName).getSlicesMap();
-      // did we find expectedSlices slices/shards?
-      if (slices.size() != expectedSlices) {
-        return "Found new collection "
-            + collectionName
-            + ", but mismatch on number of slices. Expected: "
-            + expectedSlices
-            + ", actual: "
-            + slices.size();
-      }
-      int totalShards = 0;
-      for (String sliceName : slices.keySet()) {
-        for (Replica replica : slices.get(sliceName).getReplicas()) {
-          if (nodesAllowedToRunShards != null
-              && !nodesAllowedToRunShards.contains(replica.getStr(ZkStateReader.NODE_NAME_PROP))) {
-            return "Shard "
-                + replica.getName()
-                + " created on node "
-                + replica.getNodeName()
-                + " not allowed to run shards for the created collection "
-                + collectionName;
-          }
-        }
-        totalShards += slices.get(sliceName).getReplicas().size();
-      }
-      if (totalShards != expectedTotalShards) {
-        return "Found new collection "
-            + collectionName
-            + " with correct number of slices, but mismatch on number of shards. Expected: "
-            + expectedTotalShards
-            + ", actual: "
-            + totalShards;
-      }
-      return null;
-    } else {
-      return "Could not find new collection " + collectionName;
-    }
-  }
-
-  protected void checkForCollection(
-      String collectionName,
-      List<Integer> numShardsNumReplicaList,
-      List<String> nodesAllowedToRunShards)
+  protected void checkForCollection(String collectionName, List<Integer> numShardsNumReplicaList)
       throws Exception {
     // check for an expectedSlices new collection - we poll the state
-    final TimeOut timeout = new TimeOut(120, TimeUnit.SECONDS, TimeSource.NANO_TIME);
-    boolean success = false;
-    String checkResult = "Didnt get to perform a single check";
-    while (!timeout.hasTimedOut()) {
-      checkResult =
-          checkCollectionExpectations(
-              collectionName, numShardsNumReplicaList, nodesAllowedToRunShards);
-      if (checkResult == null) {
-        success = true;
-        break;
-      }
-      Thread.sleep(500);
-    }
-    if (!success) {
-      super.printLayout();
-      fail(checkResult);
+    ZkStateReader reader = ZkStateReader.from(cloudClient);
+
+    AtomicReference<String> message = new AtomicReference<>();
+    try {
+      reader.waitForState(
+          collectionName,
+          120,
+          TimeUnit.SECONDS,
+          c -> {
+            int expectedSlices = numShardsNumReplicaList.get(0);
+            // The Math.min thing is here, because we expect replication-factor to be reduced to if
+            // there are not enough live nodes to spread all shards of a collection over different
+            // nodes.
+            int expectedShardsPerSlice = numShardsNumReplicaList.get(1);
+            int expectedTotalShards = expectedSlices * expectedShardsPerSlice;
+
+            if (c != null) {
+              Collection<Slice> slices = c.getSlices();
+              // did we find expectedSlices slices/shards?
+              if (slices.size() != expectedSlices) {
+                message.set(
+                    "Found new collection "
+                        + collectionName
+                        + ", but mismatch on number of slices. Expected: "
+                        + expectedSlices
+                        + ", actual: "
+                        + slices.size());
+                return false;
+              }
+              int totalShards = 0;
+              for (Slice slice : slices) {
+                totalShards += slice.getReplicas().size();
+              }
+              if (totalShards != expectedTotalShards) {
+                message.set(
+                    "Found new collection "
+                        + collectionName
+                        + " with correct number of slices, but mismatch on number of shards. Expected: "
+                        + expectedTotalShards
+                        + ", actual: "
+                        + totalShards);
+                return false;
+              }
+              return true;
+            } else {
+              message.set("Could not find new collection " + collectionName);
+              return false;
+            }
+          });
+    } catch (TimeoutException e) {
+      fail(message.get());
     }
   }
 
diff --git a/solr/test-framework/src/java/org/apache/solr/cloud/AbstractUnloadDistributedZkTestBase.java b/solr/test-framework/src/java/org/apache/solr/cloud/AbstractUnloadDistributedZkTestBase.java
index 2f77b69b27b..ee7fac68736 100644
--- a/solr/test-framework/src/java/org/apache/solr/cloud/AbstractUnloadDistributedZkTestBase.java
+++ b/solr/test-framework/src/java/org/apache/solr/cloud/AbstractUnloadDistributedZkTestBase.java
@@ -26,6 +26,7 @@
 import java.util.concurrent.SynchronousQueue;
 import java.util.concurrent.ThreadPoolExecutor;
 import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
 import org.apache.solr.client.solrj.SolrClient;
 import org.apache.solr.client.solrj.SolrQuery;
 import org.apache.solr.client.solrj.SolrServerException;
@@ -33,7 +34,6 @@
 import org.apache.solr.client.solrj.request.CollectionAdminRequest;
 import org.apache.solr.client.solrj.request.CoreAdminRequest.Unload;
 import org.apache.solr.common.SolrInputDocument;
-import org.apache.solr.common.cloud.DocCollection;
 import org.apache.solr.common.cloud.Replica;
 import org.apache.solr.common.cloud.Slice;
 import org.apache.solr.common.cloud.ZkCoreNodeProps;
@@ -41,12 +41,10 @@
 import org.apache.solr.common.params.ModifiableSolrParams;
 import org.apache.solr.common.util.ExecutorUtil;
 import org.apache.solr.common.util.SolrNamedThreadFactory;
-import org.apache.solr.common.util.TimeSource;
 import org.apache.solr.core.SolrCore;
 import org.apache.solr.core.SolrPaths;
 import org.apache.solr.embedded.JettySolrRunner;
 import org.apache.solr.util.TestInjection;
-import org.apache.solr.util.TimeOut;
 import org.junit.Test;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
@@ -97,42 +95,37 @@ private SolrClient newSolrClient(String url) {
   private void checkCoreNamePresenceAndSliceCount(
       String collectionName, String coreName, boolean shouldBePresent, int expectedSliceCount)
       throws Exception {
-    final TimeOut timeout = new TimeOut(45, TimeUnit.SECONDS, TimeSource.NANO_TIME);
-    Boolean isPresent = null; // null meaning "don't know"
-    while (null == isPresent || shouldBePresent != isPresent) {
-      getCommonCloudSolrClient();
-      final DocCollection docCollection =
-          cloudClient.getClusterState().getCollectionOrNull(collectionName);
-      final Collection<Slice> slices =
-          (docCollection != null) ? docCollection.getSlices() : Collections.emptyList();
-      if (timeout.hasTimedOut()) {
-        printLayout();
-        fail(
-            "checkCoreNamePresenceAndSliceCount failed:"
-                + " collection="
-                + collectionName
-                + " CoreName="
-                + coreName
-                + " shouldBePresent="
-                + shouldBePresent
-                + " isPresent="
-                + isPresent
-                + " expectedSliceCount="
-                + expectedSliceCount
-                + " actualSliceCount="
-                + slices.size());
-      }
-      if (expectedSliceCount == slices.size()) {
-        isPresent = false;
-        for (Slice slice : slices) {
-          for (Replica replica : slice.getReplicas()) {
-            if (coreName.equals(replica.get("core"))) {
-              isPresent = true;
+    ZkStateReader reader = ZkStateReader.from(cloudClient);
+    try {
+      reader.waitForState(
+          collectionName,
+          45,
+          TimeUnit.SECONDS,
+          c -> {
+            final Collection<Slice> slices = (c != null) ? c.getSlices() : Collections.emptyList();
+            if (expectedSliceCount == slices.size()) {
+              for (Slice slice : slices) {
+                for (Replica replica : slice.getReplicas()) {
+                  if (coreName.equals(replica.get("core"))) {
+                    return shouldBePresent;
+                  }
+                }
+              }
+              return !shouldBePresent;
+            } else {
+              return false;
             }
-          }
-        }
-      }
-      Thread.sleep(1000);
+          });
+    } catch (TimeoutException e) {
+      printLayout();
+      fail(
+          "checkCoreNamePresenceAndSliceCount failed:"
+              + " collection="
+              + collectionName
+              + " CoreName="
+              + coreName
+              + " shouldBePresent="
+              + shouldBePresent);
     }
   }
 

From 76cf62e58ec2f9b3430a375134960dace11ea877 Mon Sep 17 00:00:00 2001
From: Jason Gerlowski <gerlowskija@apache.org>
Date: Thu, 7 Nov 2024 06:36:52 -0800
Subject: [PATCH 32/32] Update references to ASF Jenkins jobs (#2827)

We've recently tweaked the Jenkins jobs that we run, in order to get as
many 'test' runs as possible out of our hardware.  This involved
changing some job names and details in Jenkins.

This commit updates our dev-docs and README files accordingly.
---
 README.md                 | 2 +-
 dev-docs/asf-jenkins.adoc | 7 ++++---
 2 files changed, 5 insertions(+), 4 deletions(-)

diff --git a/README.md b/README.md
index cf08fa329a5..24d419bd472 100644
--- a/README.md
+++ b/README.md
@@ -22,7 +22,7 @@ Solr is the blazing-fast, open source, multi-modal search platform built on [Apa
 It powers full-text, vector, and geospatial search at many of the world's largest organizations.
 
 [![Build Status](https://ci-builds.apache.org/job/Solr/job/Solr-Artifacts-main/badge/icon?subject=Solr%20Artifacts)](https://ci-builds.apache.org/job/Solr/job/Solr-Artifacts-main/)
-[![Build Status](https://ci-builds.apache.org/job/Solr/job/Solr-Check-main/badge/icon?subject=Solr%20Check)](https://ci-builds.apache.org/job/Solr/job/Solr-Check-main/)
+[![Build Status](https://ci-builds.apache.org/job/Solr/job/Solr-Lint-main/badge/icon?subject=Solr%20Lint)](https://ci-builds.apache.org/job/Solr/job/Solr-Lint-main/)
 
 For a complete description of the Solr project, team composition, source
 code repositories, and other details, please see the Solr web site at
diff --git a/dev-docs/asf-jenkins.adoc b/dev-docs/asf-jenkins.adoc
index b4d01bb3964..b3c59b6fd9e 100644
--- a/dev-docs/asf-jenkins.adoc
+++ b/dev-docs/asf-jenkins.adoc
@@ -10,13 +10,14 @@ This file aims to document our [ASF Jenkins](https://ci-builds.apache.org/job/So
 We run a number of jobs on Jenkins, each validating an overlapping set of concerns:
 
 * `Solr-Artifacts-*` - daily jobs that run `./gradlew assemble` to ensure that build artifacts (except docker images) can be created successfully
-* `Solr-check-*` - "hourly" jobs that run all project tests and static analysis (i.e. `test`, `integrationTest`, and `check`)
+* `Solr-Lint-*` - daily jobs that run static analysis (i.e. `precommit` and `check -x test`) on a branch
+* `Solr-Test-*` - "hourly" jobs that run all (non-integration) tests (i.e. `./gradlew test`)
+* `Solr-TestIntegration-*` - daily jobs that run project integration tests (i.e. `./gradlew integrationTests`)
 * `Solr-Docker-Nightly-*` - daily jobs that `./gradlew testDocker dockerPush` to validate docker image packaging.  Snapshot images are pushed to hub.docker.com
-* `Solr-reference-guide-*` - hourly jobs that build the Solr reference guide via `./gradlew checkSite` and push the resulting artifact to the staging/preview site `nightlies.apache.org`
+* `Solr-reference-guide-*` - daily jobs that build the Solr reference guide via `./gradlew checkSite` and push the resulting artifact to the staging/preview site `nightlies.apache.org`
 * `Solr-Smoketest-*` - daily jobs that produce a snapshot release (via the `assembleRelease` task) and run the release smoketester
 
 Most jobs that validate particular build artifacts are run "daily", which is sufficient to prevent any large breaks from creeping into the build.
-
 On the other hand, jobs that run tests are triggered "hourly" in order to squeeze as many test runs as possible out of our Jenkins hardware.
 This is a necessary consequence of Solr's heavy use of randomization in its test-suite.
 "Hourly" scheduling ensures that a test run is either currently running or in the build queue at all times, and enables us to get the maximum data points from our hardware.