You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository has been archived by the owner on Jul 20, 2022. It is now read-only.
Hello. I want to use instance emr-5.10 ( Spark 2.2 etc.)
I'll very happy if you can show me how to solve the errors.
This is my environment
In my master emr-5.10 instance I've uploaded both files:
configuration.json (original content)
full_install_jobserver_BA.sh script
and adapted this parameters in the last on:
####### EDIT THESE - START ########
JOBSERVER_VERSION="v0.8.0"
EMR_SH_FILE="s3:///jobserver/jobserver_configs/emr.sh"
EMR_CONF_FILE="s3:///jobserver/jobserver_configs/emr.conf"
S3_BUILD_ARCHIVE_DIR="s3:///jobserver/dist"
####### EDIT THESE - END ########
When I execute full_install_jobserver_BA.sh script I obtain the following error related to unresolved dependencies (complete output attached)
...
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: com.typesafe.akka#akka-slf4j_2.10;2.4.9: not found
[warn] :: com.typesafe.akka#akka-cluster_2.10;2.4.9: not found
[warn] :: io.spray#spray-routing-shapeless23_2.10;1.3.4: not found
[warn] :: com.typesafe.akka#akka-testkit_2.10;2.4.9: not found
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn]
[warn] Note: Unresolved dependencies path:
[warn] com.typesafe.akka:akka-slf4j_2.10:2.4.9 (/mnt/work/spark-jobserver/build.sbt#L11)
[warn] +- spark.jobserver:akka-app_2.10:0.8.0
[warn] com.typesafe.akka:akka-cluster_2.10:2.4.9 (/mnt/work/spark-jobserver/build.sbt#L11)
[warn] +- spark.jobserver:akka-app_2.10:0.8.0
[warn] io.spray:spray-routing-shapeless23_2.10:1.3.4 (/mnt/work/spark-jobserver/build.sbt#L11)
[warn] +- spark.jobserver:akka-app_2.10:0.8.0
[warn] com.typesafe.akka:akka-testkit_2.10:2.4.9 (/mnt/work/spark-jobserver/build.sbt#L11)
[warn] +- spark.jobserver:akka-app_2.10:0.8.0
[warn] Credentials file /root/.bintray/.credentials does not exist
[warn] Credentials file /root/.bintray/.credentials does not exist
sbt.ResolveException: unresolved dependency: com.typesafe.akka#akka-slf4j_2.10;2.4.9: not found
unresolved dependency: com.typesafe.akka#akka-cluster_2.10;2.4.9: not found
unresolved dependency: io.spray#spray-routing-shapeless23_2.10;1.3.4: not found
unresolved dependency: com.typesafe.akka#akka-testkit_2.10;2.4.9: not found
at sbt.IvyActions$.sbt$IvyActions$$resolve(IvyActions.scala:313)
at sbt.IvyActions$$anonfun$updateEither$1.apply(IvyActions.scala:191)
...
[error] (akka-app/*:update) sbt.ResolveException: unresolved dependency: com.typesafe.akka#akka-slf4j_2.10;2.4.9: not found
[error] unresolved dependency: com.typesafe.akka#akka-cluster_2.10;2.4.9: not found
[error] unresolved dependency: io.spray#spray-routing-shapeless23_2.10;1.3.4: not found
[error] unresolved dependency: com.typesafe.akka#akka-testkit_2.10;2.4.9: not found
[error] Total time: 30 s, completed 21-ene-2018 15:15:42
Assembly failed
The user-provided path /tmp/job-server/job-server.tar.gz does not exist.
tar (child): /tmp/job-server/job-server.tar.gz: No se puede open: No existe el fichero o el directorio
tar (child): Error is not recoverable: exiting now
tar: Child returned status 2
tar: Error is not recoverable: exiting now
Hello, again!!
from this link: https://aws.amazon.com/es/blogs/big-data/installing-and-running-jobserver-for-apache-spark-on-amazon-emr/
We installed Spark Job-server (SJS) component on EMR v4.x (->park 1.6 etc.)
BUT:
we need to use the new versions of EMR (v5.x -> Spark 2.2 etc.)
The question is after several days and configurations...it doesn´t work ... :(
PLEASE:
We kindly ask you for detailed instructions in order to install SJS on EMR v5.x. as well as to update this GitHub repository.
OR EVEN BETTER:
include SJS component on the list of features included on EMR Spark cluster product (Azure does that)
THANK YOU a lot, in advance!!!
Julián
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Hello. I want to use instance emr-5.10 ( Spark 2.2 etc.)
I'll very happy if you can show me how to solve the errors.
This is my environment
In my master emr-5.10 instance I've uploaded both files:
and adapted this parameters in the last on:
####### EDIT THESE - START ########
JOBSERVER_VERSION="v0.8.0"
EMR_SH_FILE="s3:///jobserver/jobserver_configs/emr.sh"
EMR_CONF_FILE="s3:///jobserver/jobserver_configs/emr.conf"
S3_BUILD_ARCHIVE_DIR="s3:///jobserver/dist"
####### EDIT THESE - END ########
In s3:///jobserver/jobserver_configs/ I've:
-emr.sh:
APP_USER=hadoop
APP_GROUP=hadoop
INSTALL_DIR=/mnt/lib/spark-jobserver
LOG_DIR=/mnt/var/log/spark-jobserver
PIDFILE=spark-jobserver.pid
JOBSERVER_MEMORY=1G
SPARK_VERSION=2.2.0
SPARK_HOME=/usr/lib/spark
SPARK_CONF_DIR=/etc/spark/conf
HADOOP_CONF_DIR=/etc/hadoop/conf
YARN_CONF_DIR=/etc/hadoop/conf
SCALA_VERSION=2.11.0
When I execute full_install_jobserver_BA.sh script I obtain the following error related to unresolved dependencies (complete output attached)
...
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: com.typesafe.akka#akka-slf4j_2.10;2.4.9: not found
[warn] :: com.typesafe.akka#akka-cluster_2.10;2.4.9: not found
[warn] :: io.spray#spray-routing-shapeless23_2.10;1.3.4: not found
[warn] :: com.typesafe.akka#akka-testkit_2.10;2.4.9: not found
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn]
[warn] Note: Unresolved dependencies path:
[warn] com.typesafe.akka:akka-slf4j_2.10:2.4.9 (/mnt/work/spark-jobserver/build.sbt#L11)
[warn] +- spark.jobserver:akka-app_2.10:0.8.0
[warn] com.typesafe.akka:akka-cluster_2.10:2.4.9 (/mnt/work/spark-jobserver/build.sbt#L11)
[warn] +- spark.jobserver:akka-app_2.10:0.8.0
[warn] io.spray:spray-routing-shapeless23_2.10:1.3.4 (/mnt/work/spark-jobserver/build.sbt#L11)
[warn] +- spark.jobserver:akka-app_2.10:0.8.0
[warn] com.typesafe.akka:akka-testkit_2.10:2.4.9 (/mnt/work/spark-jobserver/build.sbt#L11)
[warn] +- spark.jobserver:akka-app_2.10:0.8.0
[warn] Credentials file /root/.bintray/.credentials does not exist
[warn] Credentials file /root/.bintray/.credentials does not exist
sbt.ResolveException: unresolved dependency: com.typesafe.akka#akka-slf4j_2.10;2.4.9: not found
unresolved dependency: com.typesafe.akka#akka-cluster_2.10;2.4.9: not found
unresolved dependency: io.spray#spray-routing-shapeless23_2.10;1.3.4: not found
unresolved dependency: com.typesafe.akka#akka-testkit_2.10;2.4.9: not found
at sbt.IvyActions$.sbt$IvyActions$$resolve(IvyActions.scala:313)
at sbt.IvyActions$$anonfun$updateEither$1.apply(IvyActions.scala:191)
...
[error] (akka-app/*:update) sbt.ResolveException: unresolved dependency: com.typesafe.akka#akka-slf4j_2.10;2.4.9: not found
[error] unresolved dependency: com.typesafe.akka#akka-cluster_2.10;2.4.9: not found
[error] unresolved dependency: io.spray#spray-routing-shapeless23_2.10;1.3.4: not found
[error] unresolved dependency: com.typesafe.akka#akka-testkit_2.10;2.4.9: not found
[error] Total time: 30 s, completed 21-ene-2018 15:15:42
Assembly failed
The user-provided path /tmp/job-server/job-server.tar.gz does not exist.
tar (child): /tmp/job-server/job-server.tar.gz: No se puede open: No existe el fichero o el directorio
tar (child): Error is not recoverable: exiting now
tar: Child returned status 2
tar: Error is not recoverable: exiting now
Thakyou very much!!
Julián.
error.txt
The text was updated successfully, but these errors were encountered: