Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Doesn't run against opensource spark #2

Open
AndrewKL opened this issue Aug 20, 2019 · 3 comments
Open

Doesn't run against opensource spark #2

AndrewKL opened this issue Aug 20, 2019 · 3 comments

Comments

@AndrewKL
Copy link 

[hadoop@ip-10-0-61-104 ~]$ spark-submit --class com.databricks.spark.sql.perf.runExperiment --jars s3://loandrew-emr-dev/tools/spark-memory/spark-memory-core_2.11-0.1.0-SNAPSHOT.jar   --driver-java-options "-XX:MaxMetaspaceSize=200M -XX:+PrintGCDetails -Dmemory.monitor.enabled=true -Dmemory.monitor.freq=100"   --conf spark.executor.extraJavaOptions="-XX:MaxMetaspaceSize=200M -XX:+PrintGCDetails -Dmemory.monitor.enabled=true -Dmemory.monitor.freq=100"   --conf spark.executor.plugins="com.cloudera.spark.MemoryMonitorExecutorExtension"    /home/hadoop/spark-sql-perf.jar --benchmark TPCDS_2_4 --databaseName tpcds_loandrew --jobName q95 --queryNames q95 --runWarmup false    --prewarmQueryPlanning false --iterations 1 --resultLocation s3://apt-us-east-2/loandrew/result/j-KXGLEFJYBXB6/q95/d7cd6a55-e505-4212-86c5-1555ef8409a1    --tags experimentName=exchange-elimination-alias-bug-cr2-heap-logging-9,clusterId=j-KXGLEFJYBXB6    --queryOutputLocation s3://apt-us-east-2/loandrew/output/j-KXGLEFJYBXB6/q95/86e5b389-a5d0-4845-956a-b764491dbc32

NOTE the namespace difference between the two versions of ExecutorPlugin.

ccessfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 39905.
19/08/20 21:07:04 INFO NettyBlockTransferService: Server created on ip-10-0-24-240.us-east-2.compute.internal:39905
19/08/20 21:07:04 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
19/08/20 21:07:04 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(4, ip-10-0-24-240.us-east-2.compute.internal, 39905, None)
19/08/20 21:07:04 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(4, ip-10-0-24-240.us-east-2.compute.internal, 39905, None)
19/08/20 21:07:04 INFO BlockManager: external shuffle service port = 7337
19/08/20 21:07:04 INFO BlockManager: Registering executor with local external shuffle service.
19/08/20 21:07:04 INFO TransportClientFactory: Successfully created connection to ip-10-0-24-240.us-east-2.compute.internal/10.0.24.240:7337 after 1 ms (0 ms spent in bootstraps)
19/08/20 21:07:04 INFO BlockManager: Initialized BlockManager: BlockManagerId(4, ip-10-0-24-240.us-east-2.compute.internal, 39905, None)
19/08/20 21:07:04 ERROR CoarseGrainedExecutorBackend: Executor self-exiting due to : Unable to create executor due to requirement failed: com.cloudera.spark.MemoryMonitorExecutorExtension is not a subclass of org.apache.spark.ExecutorPlugin.
java.lang.IllegalArgumentException: requirement failed: com.cloudera.spark.MemoryMonitorExecutorExtension is not a subclass of org.apache.spark.ExecutorPlugin.
	at scala.Predef$.require(Predef.scala:224)
	at org.apache.spark.util.Utils$$anonfun$loadExtensions$1.apply(Utils.scala:2714)
	at org.apache.spark.util.Utils$$anonfun$loadExtensions$1.apply(Utils.scala:2711)
	at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
	at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
	at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
	at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
	at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
	at org.apache.spark.util.Utils$.loadExtensions(Utils.scala:2711)
	at org.apache.spark.executor.Executor$$anonfun$5.apply(Executor.scala:148)
	at org.apache.spark.executor.Executor$$anonfun$5.apply(Executor.scala:147)
	at org.apache.spark.util.Utils$.withContextClassLoader(Utils.scala:249)
	at org.apache.spark.executor.Executor.<init>(Executor.scala:147)
	at org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$receive$1.applyOrElse(CoarseGrainedExecutorBackend.scala:125)
	at org.apache.spark.rpc.netty.Inbox$$anonfun$process$1.apply$mcV$sp(Inbox.scala:117)
	at org.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:205)
	at org.apache.spark.rpc.netty.Inbox.process(Inbox.scala:101)
	at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:221)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
19/08/20 21:07:04 INFO CoarseGrainedExecutorBackend: MemStats: (343 of 874 heapsize(MB)(42 of 42 nonheap(MB))(874 total(MB))
19/08/20 21:07:04 INFO CoarseGrainedExecutorBackend: shuffle-server.usedHeapMemory : org.apache.spark.network.util.NettyMemoryMetrics$$Lambda$8/1123321137@778d5519
19/08/20 21:07:04 INFO CoarseGrainedExecutorBackend: shuffle-client.usedDirectMemory : org.apache.spark.network.util.NettyMemoryMetrics$$Lambda$9/685934@1573c65a
19/08/20 21:07:04 INFO CoarseGrainedExecutorBackend: shuffle-server.usedDirectMemory : org.apache.spark.network.util.NettyMemoryMetrics$$Lambda$9/685934@2946f4f
19/08/20 21:07:04 INFO CoarseGrainedExecutorBackend: shuffle-client.usedHeapMemory : org.apache.spark.network.util.NettyMemoryMetrics$$Lambda$8/1123321137@2c79f680
19/08/20 21:07:04 INFO DiskBlockManager: Shutdown hook called
19/08/20 21:07:04 INFO ShutdownHookManager: Shutdown hook called
@AndrewKL
Copy link
Author

digging through the spark-memory classes nothing seems to implement the org.apache.spark.ExecutorPlugin api which is used by the --conf spark.executor.plugins="com.cloudera.spark.MemoryMonitorExecutorExtension" parameter

@AndrewKL
Copy link
Author

i updated that class to implement both interfaces.Now I'm getting a new execution

19/08/20 21:34:20 ERROR SparkUncaughtExceptionHandler: Uncaught exception in thread Thread[dispatcher-event-loop-3,5,main]
java.lang.NoClassDefFoundError: com/quantifind/sumac/FieldArgs
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
	at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
	at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at com.cloudera.spark.MemoryMonitorArgs$.sysPropsArgs$lzycompute(MemoryMonitor.scala:404)
	at com.cloudera.spark.MemoryMonitorArgs$.sysPropsArgs(MemoryMonitor.scala:403)
	at com.cloudera.spark.MemoryMonitor$.installIfSysProps(MemoryMonitor.scala:238)
	at com.cloudera.spark.MemoryMonitorExecutorExtension.<init>(MemoryMonitor.scala:333)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.apache.spark.util.Utils$$anonfun$loadExtensions$1.apply(Utils.scala:2722)
	at org.apache.spark.util.Utils$$anonfun$loadExtensions$1.apply(Utils.scala:2711)
	at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
	at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
	at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
	at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
	at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
	at org.apache.spark.util.Utils$.loadExtensions(Utils.scala:2711)
	at org.apache.spark.executor.Executor$$anonfun$5.apply(Executor.scala:148)
	at org.apache.spark.executor.Executor$$anonfun$5.apply(Executor.scala:147)
	at org.apache.spark.util.Utils$.withContextClassLoader(Utils.scala:249)
	at org.apache.spark.executor.Executor.<init>(Executor.scala:147)
	at org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$receive$1.applyOrElse(CoarseGrainedExecutorBackend.scala:125)
	at org.apache.spark.rpc.netty.Inbox$$anonfun$process$1.apply$mcV$sp(Inbox.scala:117)
	at org.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:205)
	at org.apache.spark.rpc.netty.Inbox.process(Inbox.scala:101)
	at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:221)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: com.quantifind.sumac.FieldArgs
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 40 more
19/08/20 21:34:20 INFO CoarseGrainedExecutorBackend: MemStats: (103 of 632 heapsize(MB)(42 of 43 nonheap(MB))(632 total(MB))

@AndrewKL
Copy link
Author

AndrewKL commented Aug 21, 2019 •
edited
Loading

created pull request #4

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@AndrewKL