Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Split "SparkConfiguration" into launcher and runtime configs #41

Open
ericdill opened this issue Apr 13, 2017 · 3 comments
Open

Split "SparkConfiguration" into launcher and runtime configs #41

ericdill opened this issue Apr 13, 2017 · 3 comments

Comments

@ericdill
Copy link
Contributor

It's pretty confusing for the configuration parameters that are required for launching the spark context to be mixed in with the parameters that can be tweaked while the context is active. This is probably an API break so needs to be considered carefully

@parente
Copy link
Contributor

parente commented Apr 13, 2017

Maybe we can make the split, but retain a single global (deprecated) object that is the union of the two for a while?

@mariusvniekerk
Copy link
Collaborator

That sounds like the minimal risk approach.

@mariusvniekerk
Copy link
Collaborator

Only a subset of the spark configuration parameters actually do anything if you change them post spark context initialization

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants