Skip to content

Commit

Permalink
[LIVY-863] Missing JVM class imports for Spark3
Browse files Browse the repository at this point in the history
## What changes were proposed in this pull request?

Description of the problem: https://issues.apache.org/jira/browse/LIVY-863

The proposed fix consists of adding the missing imports that upstream Spark has when it's initiating the Java Gateway.
See the imports of Spark here: https://github.com/apache/spark/blob/87bf6b0ea4ca0618c8604895d05037edce8b7cb0/python/pyspark/java_gateway.py#L153

As far as I am aware, the java_import() does not fail or fails silently if the imported class does not exist.
But I might need to add some code to account for different Spark versions, looking for some guidance on this.

## How was this patch tested?

Tested with both valid and "bogus" imports by reviewers.
  • Loading branch information
moritzmeister authored Nov 12, 2024
1 parent 6097af1 commit 1f6bd7a
Showing 1 changed file with 3 additions and 0 deletions.
3 changes: 3 additions & 0 deletions repl/src/main/resources/fake_shell.py
Original file line number Diff line number Diff line change
Expand Up @@ -589,8 +589,11 @@ def main():
java_import(gateway.jvm, "org.apache.spark.SparkConf")
java_import(gateway.jvm, "org.apache.spark.api.java.*")
java_import(gateway.jvm, "org.apache.spark.api.python.*")
java_import(gateway.jvm, "org.apache.spark.ml.python.*")
java_import(gateway.jvm, "org.apache.spark.mllib.api.python.*")
java_import(gateway.jvm, "org.apache.spark.resource.*")
java_import(gateway.jvm, "org.apache.spark.sql.*")
java_import(gateway.jvm, "org.apache.spark.sql.api.python.*")
java_import(gateway.jvm, "org.apache.spark.sql.hive.*")
java_import(gateway.jvm, "scala.Tuple2")

Expand Down

0 comments on commit 1f6bd7a

Please sign in to comment.