Jetty12 + EE10 #2876

iamsanjay · 2024-11-19T17:34:55Z

https://issues.apache.org/jira/browse/SOLR-17069

Currently, Jetty supports several environments:

EE8 (Jetty10): Implements the Java EE8 specifications, supporting javax.servlet.
EE9 (Jetty11): Implements Jakarta EE9 specifications, introducing the namespace migration from javax.* to jakarta.*.
EE10 (Jetty12): Implements Jakarta EE10 specifications, including the Jakarta Servlet 6.0 specification, fully adopting the jakarta.* namespace.

Jetty 12 providing ongoing support for both older and upcoming EE specifications [from https://github.com/jetty/jetty.project/issues/10485]

In my opinion, Jetty12 + EE8 is the best next step. It retains compatibility with javax.servlet without forcing the removal of modules that haven’t transitioned to jakarta.* yet, while benefiting from all the new features introduced in Jetty12.

#1509

risdenk · 2024-11-19T17:46:26Z

This should link to https://issues.apache.org/jira/browse/SOLR-17069 and potentially #1509

gradle/libs.versions.toml

risdenk · 2024-11-19T17:47:21Z

gradle/libs.versions.toml

@@ -305,7 +306,7 @@ commonsio-commonsio = { module = "commons-io:commons-io", version.ref = "commons
 cybozulabs-langdetect = { module = "com.cybozu.labs:langdetect", version.ref = "cybozulabs-langdetect" }
 dropwizard-metrics-core = { module = "io.dropwizard.metrics:metrics-core", version.ref = "dropwizard-metrics" }
 dropwizard-metrics-graphite = { module = "io.dropwizard.metrics:metrics-graphite", version.ref = "dropwizard-metrics" }
-dropwizard-metrics-jetty10 = { module = "io.dropwizard.metrics:metrics-jetty10", version.ref = "dropwizard-metrics" }
+dropwizard-metrics-jetty10 = { module = "io.dropwizard.metrics:metrics-jetty12-ee10", version.ref = "dropwizard-metrics" }


does it matter that this is ee10 vs ee8 like the description?

also the key should change from dropwizard-metrics-jetty10

risdenk · 2024-11-19T17:49:02Z

solr/core/src/java/org/apache/solr/handler/admin/ShowFileRequestHandler.java

    KNOWN_MIME_TYPES.add("text/xml");
    KNOWN_MIME_TYPES.add("text/javascript");
+    KNOWN_MIME_TYPES.add("text/csv");


Why did csv/xhtml get added?

Starting from Jetty 11 these two mime types are not included in MimeTypes.Type, but we have tests that check if they are supported. Not sure if this could cause any issues with jetty though.

solr/core/src/test/org/apache/solr/cloud/OverseerStatusTest.java

solr/licenses/jetty-ee10-servlet-12.0.10.jar.sha1

solr/test-framework/src/java/org/apache/solr/embedded/JettySolrRunner.java

risdenk · 2024-11-19T17:52:24Z

solr/test-framework/src/java/org/apache/solr/embedded/JettySolrRunner.java

    }

    chain = injectJettyHandlers(chain);

    if (config.enableV2) {
      RewriteHandler rwh = new RewriteHandler();
      rwh.setHandler(chain);
-      rwh.setRewriteRequestURI(true);
-      rwh.setRewritePathInfo(false);
+      // rwh.setRewriteRequestURI(true);


commented out?

iamsanjay · 2024-11-20T12:50:32Z

s3-repository test cases started to failing as HandlerWrapper class not available anymore which was part of Jetty10 and now with Jetty12, it's part of ee8.nested.
However, spring-boot jetty-starter doesn't have anything for ee8. They do have support for ee10 webapps, and adding those runs the test successfully but there are multiple library conflict is there. Working on it.

risdenk · 2024-11-20T13:33:48Z

s3mock was a problem when I last looked at Jetty 11/12 due to it needing to stay on 2.x due to Java 17 required in 3.x. However, now that we are on JDK 17 on main, we should be able to upgrade to s3mock 3.x - https://github.com/adobe/S3Mock/releases/tag/3.0.0 (latest being https://github.com/adobe/S3Mock/releases/tag/3.11.0)

I wonder if that will address the Jetty incompatibilities since this upgraded Spring as well.

This upgrade of s3mock should be done separately from any Jetty upgrade if possible.

malliaridis · 2024-11-19T20:08:06Z

gradle/libs.versions.toml

@@ -138,7 +139,7 @@ jayway-jsonpath = "2.9.0"
 jctools = "4.0.5"
 jersey = "3.1.9"
 # TODO Sync with jersey versions
-jersey-containers = "2.39.1"
+jersey-containers = "3.1.0"


I would go with jersey version 3.1.9 if possible, so that we can remove this entry completely (see line above).

gradle/libs.versions.toml

malliaridis · 2024-11-19T20:14:13Z

solr/core/src/java/org/apache/solr/handler/admin/ShowFileRequestHandler.java

    KNOWN_MIME_TYPES.add("text/xml");
    KNOWN_MIME_TYPES.add("text/javascript");
+    KNOWN_MIME_TYPES.add("text/csv");


Starting from Jetty 11 these two mime types are not included in MimeTypes.Type, but we have tests that check if they are supported. Not sure if this could cause any issues with jetty though.

solr/core/src/test/org/apache/solr/util/tracing/TestHttpServletRequestGetter.java

iamsanjay · 2024-11-22T14:07:19Z

For S3Mock, the library is upgraded to support Jetty12. Test cases are running now! For s3mock to deal in separate PR, I tried to maintain two different set of jetty(10 & 12), but there were conflict errors. So for now I just upgraded s3Mock libs to also use Jetty12.

Facing some issues with gcs-repository. Running successfully as Intellij test but failing If run via gradlew.

./gradlew :solr:modules:gcs-repository:test --tests "org.apache.solr.gcs.GCSIncrementalBackupTest" -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=90E261AA14A5CB20 -Ptests.timeoutSuite=600000! -Ptests.file.encoding=US-ASCII --debug

 java.lang.IllegalStateException: Unable to locate keystore resource file in classpath: SSLTestConfig.hostname-and-ip-missmatch.keystore
        at __randomizedtesting.SeedInfo.seed([90E261AA14A5CB20]:0)
        at org.apache.solr.util.SSLTestConfig.<init>(SSLTestConfig.java:121)
        at org.apache.solr.util.SSLTestConfig.<init>(SSLTestConfig.java:83)
        at org.apache.solr.util.RandomizeSSL$SSLRandomizer.createSSLTestConfig(RandomizeSSL.java:129)
        at org.apache.solr.SolrTestCaseJ4.buildSSLConfig(SolrTestCaseJ4.java:475)
        at org.apache.solr.SolrTestCaseJ4.setupTestCases(SolrTestCaseJ4.java:298)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
        at java.base/java.lang.reflect.Method.invoke(Method.java:580)

I did changed some code in SSLTestConfig

 - trustStore = keyStore = Resource.newClassPathResource(resourceName);
 + trustStore = keyStore = ResourceFactory.root().newSystemResource(resourceName);

dsmiley · 2024-11-24T21:27:18Z

The discussion here relating to S3 dependencies reminds me of Lucene's PR apache/lucene#13949 (comment) with a comment yesterday on use of "aws-lightweight-client-java"

iamsanjay · 2025-01-17T06:16:47Z

Most of the test cases, I believe, are failing due to some issue with the Restore Command. Now in the logs, there is not much details regarding the issue and the only thing I got is.

{responseHeader={status=0, QTime=611}, STATUS=failed, response=Error handling 'RESTORECORE' action}

But If I try to debug I can see more elaborated exception message which I believe for some reason was suppressed.

Cannot invoke "org.eclipse.jetty.server.Request.getAttribute(String)" because the return value of "org.eclipse.jetty.ee10.servlet.ServletApiRequest.getRequest()" is null

There is somewhat related issue opened on Jetty's github jetty/jetty.project#12518

iamsanjay · 2025-01-19T07:35:29Z

Got some of the commits from the main branch. It would be great to remove those before someone would review this PR again.

iamsanjay · 2025-01-20T11:06:51Z

The only failing test is the flaky one, I checked on develocity.

Reproduce with: ./gradlew :solr:modules:ltr:test --tests "org.apache.solr.ltr.TestLTRQParserPlugin.ltr_expensiveFeatureRescoringAndPartialResultsNotTolerated_shouldRaiseException" "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=690CA0CDF5423C59 -Ptests.timeoutSuite=600000! -Ptests.file.encoding=UTF-8

dsmiley

Did you read through Jetty's release notes to see if there's stuff we should do that would otherwise not be apparen't if we only fight imports & dependencies?

BTW I have yet to review the last 6 files but GitHub (or Intellij?) is struggling right now...

dsmiley · 2025-01-28T06:40:20Z

solr/core/src/java/org/apache/solr/handler/admin/api/CoreAdminAPIBase.java

+      try {
+        TraceUtils.setDbInstance(req, coreName);
+      } catch (Exception e) {
+        // TODO: Find better way to fix NPE


we use "nocommit" which will also fail precommit until we get it fixed, serving as a reminder. Any way, this try-catch definitely shouldn't be here if it should be anywhere. We should fix what's null instead.

dsmiley · 2025-01-28T06:46:36Z

solr/core/src/test/org/apache/solr/cloud/CloudExitableDirectoryReaderTest.java

@@ -106,7 +106,7 @@ public static void setupCluster() throws Exception {
          ((Metered)
                  metricRegistry
                      .getMetrics()
-                      .get("org.eclipse.jetty.servlet.ServletContextHandler.2xx-responses"))
+                      .get("org.eclipse.jetty.server.Handler$Wrapper.2xx-responses"))


This metrics name seems poor. I wonder if it's due to us/Solr (thus we have easy control) or is this name chosen by Jetty / DropWizard.

dsmiley · 2025-01-28T07:02:19Z

solr/core/src/test/org/apache/solr/servlet/HttpSolrCallCloudTest.java

@@ -93,40 +114,66 @@ private void assertCoreChosen(int numCores, TestRequest testRequest) throws Unav
    assertEquals(numCores, coreNames.size());
  }

-  private static class TestResponse extends Response {
+  public static class TestRequest implements HttpServletRequest {


I hate mocks but as I look at this, mocking seems perfect so you don't have to implement everything. Or maybe there's a base class as there was before that you can extend. (Same feedback for another source file you did this in)

dsmiley · 2025-01-28T07:14:18Z

solr/core/src/test/org/apache/solr/util/tracing/TestHttpServletRequestGetter.java

@@ -40,8 +49,9 @@ public void test() {
            "b", Set.of("1"),
            "c", Set.of("2"));

+    HttpServletRequest httpServletRequestMock = mock(HttpServletRequest.class);


aha; you figured out mocking :-)

dsmiley · 2025-01-28T07:15:39Z

solr/cross-dc-manager/src/java/org/apache/solr/crossdc/manager/consumer/Consumer.java

@@ -21,8 +21,8 @@
 import static org.apache.solr.crossdc.common.KafkaCrossDcConf.ZK_CONNECT_STRING;

 import com.codahale.metrics.SharedMetricRegistries;
-import com.codahale.metrics.servlets.MetricsServlet;
-import com.codahale.metrics.servlets.ThreadDumpServlet;
+import io.dropwizard.metrics.servlets.MetricsServlet;


it's suspicious to see we import com.codahale & io.dropwizard. Maybe okay but I wasn't expecting.

dsmiley · 2025-01-28T07:17:05Z

solr/modules/s3-repository/build.gradle

@@ -20,7 +20,9 @@ apply plugin: 'java-library'
 description = 'S3 Repository'

 dependencies {
-  api project(':solr:core')
+  api(project(':solr:core')) {
+    exclude group: 'org.ow2.asm', module: '*'


them solr:core shouldn't be publishing that dependency in the first place. At least see if that can be done.

dsmiley · 2025-01-28T07:20:09Z

solr/server/etc/jetty.xml

@@ -214,14 +212,16 @@
          <Set name="contexts">
            <Ref refid="Contexts" />
          </Set>
+          <!--


appears to simply be some useless metadata we can remove

dsmiley · 2025-01-28T07:22:57Z

solr/solrj/src/test/org/apache/solr/client/solrj/embedded/JettyWebappTest.java

-        new DefaultSessionIdManager(server, new Random(random().nextLong())));
-    new WebAppContext(server, path, "/solr");
+    server.addBean(new DefaultSessionIdManager(server, new Random(random().nextLong())));
+    var webAppContext = new WebAppContext(path, "/solr");


minor: inline

dsmiley · 2025-01-28T07:24:25Z

solr/test-framework/build.gradle

@@ -65,16 +65,19 @@ dependencies {
  implementation libs.apache.log4j.api
  implementation libs.apache.log4j.core
  implementation libs.dropwizard.metrics.core
-  implementation libs.dropwizard.metrics.jetty10
+  implementation (libs.dropwizard.metrics.jetty12.ee10){
+    exclude group: 'jakarta.servlet', module: 'jakarta.servlet-api'


as long as we explicitly depend on libs.jakarta.servlet.api below (we do), I wonder do we really need to exclude it here?

dsmiley · 2025-01-28T20:06:06Z

solr/test-framework/src/java/org/apache/solr/embedded/JettySolrRunner.java

-      chain = root;
+      chain.setHandler(root);


it's be clearer to do this at the front of this block.

github-actions bot added dependencies Dependency upgrades test-framework jetty-server client:solrj module:hadoop-auth module:jwt-auth tool:build tests cat:search cat:cloud cat:index cat:cli cat:schema cat:security labels Nov 19, 2024

iamsanjay requested a review from risdenk November 19, 2024 17:52

risdenk reviewed Nov 19, 2024

View reviewed changes

malliaridis reviewed Nov 20, 2024

View reviewed changes

github-actions bot added the module:s3-repository label Nov 22, 2024

github-actions bot added documentation Improvements or additions to documentation cat:api cat:packagemanager labels Nov 27, 2024

iamsanjay force-pushed the jetty12_ee8 branch from e321ba0 to 31e58a1 Compare November 27, 2024 04:23

github-actions bot removed documentation Improvements or additions to documentation cat:api cat:packagemanager labels Nov 27, 2024

iamsanjay changed the base branch from main to branch_9_8 January 19, 2025 06:38

iamsanjay changed the base branch from branch_9_8 to main January 19, 2025 06:38

iamsanjay marked this pull request as ready for review January 19, 2025 07:35

iamsanjay added 17 commits January 20, 2025 15:40

Jetty12 + EE8 \u1F680

421c095

Update Jetty deps for s3mock

0b80a80

add missing licenses

ed63623

adding permissions to solr test policy

aecec4b

Fix TestHttpServletRequestGetter

1629241

tidy

52d5416

jetty xml changes

a047f5e

versions.lock updated

d98b23d

Set DefaultSessionIdManager on Server

7025003

removed extra logging annotation

221c3f5

workaround for removing headers

c813d4a

versions.lock

873bf7d

ee10 switch

e4e4540

Fix a metric test

fdaf430

Fix JettyWebappTest

871b6c4

Fix NPE request.getAttribute

780c0a2

rebase branch

dbe7c4f

iamsanjay force-pushed the jetty12_ee8 branch from a30414d to dbe7c4f Compare January 20, 2025 10:43

tidy

4df91b2

iamsanjay changed the title ~~[WIP] Jetty12 + EE10~~ Jetty12 + EE10 Jan 20, 2025

iamsanjay requested review from risdenk and dsmiley January 20, 2025 14:45

dsmiley reviewed Jan 28, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Jetty12 + EE10 #2876

Jetty12 + EE10 #2876

iamsanjay commented Nov 19, 2024 •

edited

Loading

risdenk commented Nov 19, 2024

risdenk Nov 19, 2024

risdenk Nov 19, 2024

risdenk Nov 19, 2024

malliaridis Nov 19, 2024

risdenk Nov 19, 2024

iamsanjay commented Nov 20, 2024

risdenk commented Nov 20, 2024

malliaridis Nov 19, 2024

malliaridis Nov 19, 2024

iamsanjay commented Nov 22, 2024

dsmiley commented Nov 24, 2024

iamsanjay commented Jan 17, 2025 •

edited

Loading

iamsanjay commented Jan 19, 2025

iamsanjay commented Jan 20, 2025

dsmiley left a comment

dsmiley Jan 28, 2025

dsmiley Jan 28, 2025

dsmiley Jan 28, 2025

dsmiley Jan 28, 2025

dsmiley Jan 28, 2025

dsmiley Jan 28, 2025

dsmiley Jan 28, 2025

dsmiley Jan 28, 2025

dsmiley Jan 28, 2025

dsmiley Jan 28, 2025

Jetty12 + EE10 #2876

Are you sure you want to change the base?

Jetty12 + EE10 #2876

Conversation

iamsanjay commented Nov 19, 2024 • edited Loading

risdenk commented Nov 19, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

iamsanjay commented Nov 20, 2024

risdenk commented Nov 20, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

iamsanjay commented Nov 22, 2024

dsmiley commented Nov 24, 2024

iamsanjay commented Jan 17, 2025 • edited Loading

iamsanjay commented Jan 19, 2025

iamsanjay commented Jan 20, 2025

dsmiley left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

iamsanjay commented Nov 19, 2024 •

edited

Loading

iamsanjay commented Jan 17, 2025 •

edited

Loading