Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Issue with basic auth #672

Open
sgowda97 opened this issue Jan 3, 2024 · 3 comments
Open

Issue with basic auth #672

sgowda97 opened this issue Jan 3, 2024 · 3 comments

Comments

@sgowda97
Copy link

sgowda97 commented Jan 3, 2024 •
edited
Loading

Hi,

I am running solr-operator 0.8.0, solr 9.4.0 on GKE. Here is my YAML file

apiVersion: solr.apache.org/v1beta1
kind: SolrCloud
metadata:
  name: dev
spec:
  solrAddressability:
      commonServicePort: 80
      external:
        domainName: HOST
        method: Ingress
        nodePortOverride: 80
        useExternalAddress: false
      podPort: 8983
  customSolrKubeOptions:
    podOptions:
      resources:
        limits:
          memory: 512Mi
          cpu: 500m
        requests:
          cpu: 500m
          memory: 256Mi
  dataStorage:
    persistent:
      pvcTemplate:
        spec:
          resources:
            requests:
              storage: 10Gi
      reclaimPolicy: Delete
  replicas: 2
  solrImage:
    repository: solr
    tag: "9.4.0"
  solrJavaMem: -Xms256M -Xmx256M
  updateStrategy:
    method: StatefulSet
  zookeeperRef:
    provided:
      chroot: /dev
      image:
        pullPolicy: IfNotPresent
        repository: pravega/zookeeper
        tag: 0.2.8
      persistence:
        reclaimPolicy: Delete
        spec:
          accessModes:
          - ReadWriteOnce
          resources:
            requests:
              storage: 5Gi
      replicas: 1
      zookeeperPodPolicy:
        resources:
          limits:
            memory: 256Mi
            cpu: 500m
          requests:
            cpu: 500m
            memory: 128Mi
  solrSecurity:
      authenticationType: Basic

The pods keep crashing. To inspect, I logged one of the pod. This is the error I am getting;

$ ksolr logs pod/dev-solrcloud-1 -f
Defaulted container "solrcloud-node" out of: solrcloud-node, cp-solr-xml (init), setup-zk (init)
Starting Solr
Java 17 detected. Enabled workaround for SOLR-16463
[0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
WARNING: A command line option has enabled the Security Manager
WARNING: The Security Manager is deprecated and will be removed in a future release
2024-01-03 07:27:01.682 INFO  (main) [] o.e.j.s.Server jetty-10.0.17; built: 2023-10-09T18:22:21.150Z; git: af15f12297adf5c5083e1f2f8f4c5974438bca25; jvm 17.0.9+9
2024-01-03 07:27:04.075 WARN  (main) [] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2024-01-03 07:27:04.279 INFO  (main) [] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2024-01-03 07:27:04.369 INFO  (main) [] o.a.s.s.CoreContainerProvider  ___      _       Welcome to Apache Solr™ version 9.4.0
2024-01-03 07:27:04.370 INFO  (main) [] o.a.s.s.CoreContainerProvider / __| ___| |_ _   Starting in cloud mode on port 8983
2024-01-03 07:27:04.370 INFO  (main) [] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_|  Install dir: /opt/solr-9.4.0
2024-01-03 07:27:04.370 INFO  (main) [] o.a.s.s.CoreContainerProvider |___/\___/_|_|    Start time: 2024-01-03T07:27:04.370560543Z
2024-01-03 07:27:04.376 INFO  (main) [] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_13.log
2024-01-03 07:27:04.376 INFO  (main) [] o.a.s.s.CoreContainerProvider Log level override, property solr.log.level=INFO
2024-01-03 07:27:04.570 INFO  (main) [] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2024-01-03 07:27:04.687 WARN  (main) [] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2024-01-03 07:27:04.967 INFO  (main) [] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
2024-01-03 07:27:05.067 INFO  (zkConnectionManagerCallback-2-thread-1) [] o.a.s.c.c.ConnectionManager zkClient has connected
2024-01-03 07:27:05.069 INFO  (main) [] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2024-01-03 07:27:05.069 WARN  (main) [] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2024-01-03 07:27:05.188 INFO  (main) [] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
2024-01-03 07:27:05.189 INFO  (main) [] o.a.s.c.SolrXmlConfig Loading solr.xml from /var/solr/data/solr.xml
2024-01-03 07:27:05.387 INFO  (main) [] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.4.0/lib]
2024-01-03 07:27:10.179 INFO  (main) [] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2024-01-03 07:27:11.977 WARN  (main) [] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2024-01-03 07:27:11.977 INFO  (main) [] o.a.s.c.ZkContainer Zookeeper client=dev-solrcloud-zookeeper-0.dev-solrcloud-zookeeper-headless.solr.svc.cluster.local:2181/dev
2024-01-03 07:27:12.072 WARN  (main) [] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2024-01-03 07:27:12.075 INFO  (main) [] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2024-01-03 07:27:12.087 INFO  (zkConnectionManagerCallback-12-thread-1) [] o.a.s.c.c.ConnectionManager zkClient has connected
2024-01-03 07:27:12.088 INFO  (main) [] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2024-01-03 07:27:12.088 WARN  (main) [] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2024-01-03 07:27:12.275 INFO  (main) [] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2024-01-03 07:27:12.288 INFO  (main) [] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2024-01-03 07:27:12.366 INFO  (zkConnectionManagerCallback-14-thread-1) [] o.a.s.c.c.ConnectionManager zkClient has connected
2024-01-03 07:27:12.366 INFO  (main) [] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2024-01-03 07:27:12.778 WARN  (main) [] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2024-01-03 07:27:12.969 INFO  (main) [] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2024-01-03 07:27:13.067 INFO  (main) [] o.a.s.c.OverseerElectionContext I am going to be the leader dev-solrcloud-1.solr:80_solr
2024-01-03 07:27:13.080 INFO  (main) [] o.a.s.c.Overseer Overseer (id=72063280979050514-dev-solrcloud-1.solr:80_solr-n_0000000000) starting
2024-01-03 07:27:13.585 INFO  (OverseerStateUpdate-72063280979050514-dev-solrcloud-1.solr:80_solr-n_0000000000) [] o.a.s.c.Overseer Starting to work on the main queue : dev-solrcloud-1.solr:80_solr
2024-01-03 07:27:13.668 INFO  (main) [] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/dev-solrcloud-1.solr:80_solr
2024-01-03 07:27:13.770 INFO  (zkCallback-13-thread-1) [] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2024-01-03 07:27:14.079 ERROR (main) [] o.a.s.s.CoreContainerProvider Could not start Solr. Check solr/home property and the logs => java.lang.NullPointerException: Cannot invoke "java.util.Map.get(Object)" because the return value of "org.apache.solr.handler.admin.SecurityConfHandler$SecurityConfig.getData()" is null
	at org.apache.solr.core.CoreContainer.reloadSecurityProperties(CoreContainer.java:1186)
java.lang.NullPointerException: Cannot invoke "java.util.Map.get(Object)" because the return value of "org.apache.solr.handler.admin.SecurityConfHandler$SecurityConfig.getData()" is null
	at org.apache.solr.core.CoreContainer.reloadSecurityProperties(CoreContainer.java:1186) ~[?:?]
	at org.apache.solr.core.CoreContainer.loadInternal(CoreContainer.java:856) ~[?:?]
	at org.apache.solr.core.CoreContainer.load(CoreContainer.java:765) ~[?:?]
	at org.apache.solr.servlet.CoreContainerProvider.createCoreContainer(CoreContainerProvider.java:427) ~[?:?]
	at org.apache.solr.servlet.CoreContainerProvider.init(CoreContainerProvider.java:246) ~[?:?]
	at org.apache.solr.servlet.CoreContainerProvider.contextInitialized(CoreContainerProvider.java:116) ~[?:?]
	at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1049) ~[jetty-server-10.0.17.jar:10.0.17]
	at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:624) ~[jetty-servlet-10.0.17.jar:10.0.17]
	at org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:984) ~[jetty-server-10.0.17.jar:10.0.17]
	at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:740) ~[jetty-servlet-10.0.17.jar:10.0.17]
	at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:392) ~[jetty-servlet-10.0.17.jar:10.0.17]
	at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1304) ~[jetty-webapp-10.0.17.jar:10.0.17]
	at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:901) ~[jetty-server-10.0.17.jar:10.0.17]
	at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:306) ~[jetty-servlet-10.0.17.jar:10.0.17]
	at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:532) ~[jetty-webapp-10.0.17.jar:10.0.17]
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93) ~[jetty-util-10.0.17.jar:10.0.17]
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:171) ~[jetty-util-10.0.17.jar:10.0.17]
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:121) ~[jetty-util-10.0.17.jar:10.0.17]
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:89) ~[jetty-server-10.0.17.jar:10.0.17]
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93) ~[jetty-util-10.0.17.jar:10.0.17]
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:171) ~[jetty-util-10.0.17.jar:10.0.17]
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114) ~[jetty-util-10.0.17.jar:10.0.17]
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:89) ~[jetty-server-10.0.17.jar:10.0.17]
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93) ~[jetty-util-10.0.17.jar:10.0.17]
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:171) ~[jetty-util-10.0.17.jar:10.0.17]
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:121) ~[jetty-util-10.0.17.jar:10.0.17]
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:89) ~[jetty-server-10.0.17.jar:10.0.17]
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93) ~[jetty-util-10.0.17.jar:10.0.17]
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:171) ~[jetty-util-10.0.17.jar:10.0.17]
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114) ~[jetty-util-10.0.17.jar:10.0.17]
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:89) ~[jetty-server-10.0.17.jar:10.0.17]
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93) ~[jetty-util-10.0.17.jar:10.0.17]
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:171) ~[jetty-util-10.0.17.jar:10.0.17]
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114) ~[jetty-util-10.0.17.jar:10.0.17]
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:89) ~[jetty-server-10.0.17.jar:10.0.17]
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:221) ~[jetty-server-10.0.17.jar:10.0.17]
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93) ~[jetty-util-10.0.17.jar:10.0.17]
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:171) ~[jetty-util-10.0.17.jar:10.0.17]
	at org.eclipse.jetty.server.Server.start(Server.java:470) ~[jetty-server-10.0.17.jar:10.0.17]
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114) ~[jetty-util-10.0.17.jar:10.0.17]
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:89) ~[jetty-server-10.0.17.jar:10.0.17]
	at org.eclipse.jetty.server.Server.doStart(Server.java:415) ~[jetty-server-10.0.17.jar:10.0.17]
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93) ~[jetty-util-10.0.17.jar:10.0.17]
	at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1919) ~[jetty-xml-10.0.17.jar:10.0.17]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
	at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
	at org.eclipse.jetty.start.Main.invokeMain(Main.java:229) ~[start.jar:10.0.17]
	at org.eclipse.jetty.start.Main.start(Main.java:528) ~[start.jar:10.0.17]
	at org.eclipse.jetty.start.Main.main(Main.java:76) ~[start.jar:10.0.17]
2024-01-03 07:27:14.171 ERROR (main) [] o.a.s.s.CoreContainerProvider Error processing the request. CoreContainer is either not initialized or shutting down.
2024-01-03 07:27:14.171 ERROR (main) [] o.a.s.s.SolrDispatchFilter Could not start Dispatch Filter. => javax.servlet.UnavailableException: Error processing the request. CoreContainer is either not initialized or shutting down.
	at org.apache.solr.servlet.CoreContainerProvider.waitForCoreContainer(CoreContainerProvider.java:156)

This does not happen if I kubectl apply -f yaml.yaml without the basic auth property. One more thing I noticed is, the error miraculously goes away if I just kubectl delete pod/dev-solrcloud-1 since it gets re-created.
@HoustonPutman
Copy link
Contributor

That sounds like a race-condition for bootstrapping the security.json... How strange. This only happens when creating the cloud for the first time, correct?

@sgowda97
Copy link
Author

@HoustonPutman Yes, happens only for the first time. As a workaround, I added the following code to my solr-cloud yaml, it temporarily fixed the issue.

initContainers: # fix for https://github.com/apache/solr-operator/issues/672#issue-2063779008
      - command:
        - sh
        - -c
        - "host=$(printenv | grep ZOOKEEPER_CLIENT_SERVICE_HOST | cut -d= -f2); until echo 'stat' | nc $host 2181 | grep Zookeeper; do echo waiting for zookeeper; sleep 2; done;"
        image: library/busybox:1.28.0-glibc
        imagePullPolicy: IfNotPresent
        name: verify-zookeeper-before-solr

@VitalijusDBentley
Copy link

Also possible solution is to release with replicas: 1 and after it is set-up change to desired replica count. Also @HoustonPutman maybe solr-operator could migrate from initContainers to Jobs for set-ups and upgrades of the cluster. Especially some operations requires Solr shutdown before applying changes. This would allow solr-operator to handle these kind of operations as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@HoustonPutman @sgowda97 @VitalijusDBentley