Skip to content
This repository has been archived by the owner on Jul 20, 2022. It is now read-only.

Error using emr-5.10.0 instance #93

Open
jgomezcuadrado opened this issue Jan 21, 2018 · 1 comment
Open

Error using emr-5.10.0 instance #93

jgomezcuadrado opened this issue Jan 21, 2018 · 1 comment

Comments

@jgomezcuadrado
Copy link

Hello. I want to use instance emr-5.10 ( Spark 2.2 etc.)
I'll very happy if you can show me how to solve the errors.
This is my environment
In my master emr-5.10 instance I've uploaded both files:

  • configuration.json (original content)
  • full_install_jobserver_BA.sh script
    and adapted this parameters in the last on:
    ####### EDIT THESE - START ########
    JOBSERVER_VERSION="v0.8.0"
    EMR_SH_FILE="s3:///jobserver/jobserver_configs/emr.sh"
    EMR_CONF_FILE="s3:///jobserver/jobserver_configs/emr.conf"
    S3_BUILD_ARCHIVE_DIR="s3:///jobserver/dist"
    ####### EDIT THESE - END ########

In s3:///jobserver/jobserver_configs/ I've:

-emr.sh:
APP_USER=hadoop
APP_GROUP=hadoop
INSTALL_DIR=/mnt/lib/spark-jobserver
LOG_DIR=/mnt/var/log/spark-jobserver
PIDFILE=spark-jobserver.pid
JOBSERVER_MEMORY=1G
SPARK_VERSION=2.2.0
SPARK_HOME=/usr/lib/spark
SPARK_CONF_DIR=/etc/spark/conf
HADOOP_CONF_DIR=/etc/hadoop/conf
YARN_CONF_DIR=/etc/hadoop/conf
SCALA_VERSION=2.11.0

  • emr_contexts.conf (github original content)

When I execute full_install_jobserver_BA.sh script I obtain the following error related to unresolved dependencies (complete output attached)
...
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: com.typesafe.akka#akka-slf4j_2.10;2.4.9: not found
[warn] :: com.typesafe.akka#akka-cluster_2.10;2.4.9: not found
[warn] :: io.spray#spray-routing-shapeless23_2.10;1.3.4: not found
[warn] :: com.typesafe.akka#akka-testkit_2.10;2.4.9: not found
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn]
[warn] Note: Unresolved dependencies path:
[warn] com.typesafe.akka:akka-slf4j_2.10:2.4.9 (/mnt/work/spark-jobserver/build.sbt#L11)
[warn] +- spark.jobserver:akka-app_2.10:0.8.0
[warn] com.typesafe.akka:akka-cluster_2.10:2.4.9 (/mnt/work/spark-jobserver/build.sbt#L11)
[warn] +- spark.jobserver:akka-app_2.10:0.8.0
[warn] io.spray:spray-routing-shapeless23_2.10:1.3.4 (/mnt/work/spark-jobserver/build.sbt#L11)
[warn] +- spark.jobserver:akka-app_2.10:0.8.0
[warn] com.typesafe.akka:akka-testkit_2.10:2.4.9 (/mnt/work/spark-jobserver/build.sbt#L11)
[warn] +- spark.jobserver:akka-app_2.10:0.8.0
[warn] Credentials file /root/.bintray/.credentials does not exist
[warn] Credentials file /root/.bintray/.credentials does not exist
sbt.ResolveException: unresolved dependency: com.typesafe.akka#akka-slf4j_2.10;2.4.9: not found
unresolved dependency: com.typesafe.akka#akka-cluster_2.10;2.4.9: not found
unresolved dependency: io.spray#spray-routing-shapeless23_2.10;1.3.4: not found
unresolved dependency: com.typesafe.akka#akka-testkit_2.10;2.4.9: not found
at sbt.IvyActions$.sbt$IvyActions$$resolve(IvyActions.scala:313)
at sbt.IvyActions$$anonfun$updateEither$1.apply(IvyActions.scala:191)
...
[error] (akka-app/*:update) sbt.ResolveException: unresolved dependency: com.typesafe.akka#akka-slf4j_2.10;2.4.9: not found
[error] unresolved dependency: com.typesafe.akka#akka-cluster_2.10;2.4.9: not found
[error] unresolved dependency: io.spray#spray-routing-shapeless23_2.10;1.3.4: not found
[error] unresolved dependency: com.typesafe.akka#akka-testkit_2.10;2.4.9: not found
[error] Total time: 30 s, completed 21-ene-2018 15:15:42
Assembly failed

The user-provided path /tmp/job-server/job-server.tar.gz does not exist.
tar (child): /tmp/job-server/job-server.tar.gz: No se puede open: No existe el fichero o el directorio
tar (child): Error is not recoverable: exiting now
tar: Child returned status 2
tar: Error is not recoverable: exiting now

Thakyou very much!!
Julián.

error.txt

@jgomezcuadrado
Copy link
Author

Hello, again!!
from this link:
https://aws.amazon.com/es/blogs/big-data/installing-and-running-jobserver-for-apache-spark-on-amazon-emr/
We installed Spark Job-server (SJS) component on EMR v4.x (->park 1.6 etc.)
BUT:
we need to use the new versions of EMR (v5.x -> Spark 2.2 etc.)
The question is after several days and configurations...it doesn´t work ... :(
PLEASE:
We kindly ask you for detailed instructions in order to install SJS on EMR v5.x. as well as to update this GitHub repository.
OR EVEN BETTER:
include SJS component on the list of features included on EMR Spark cluster product (Azure does that)

THANK YOU a lot, in advance!!!
Julián

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant