refactor the process of controlling a training job #39
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
m3ngyang
commented
Jul 12, 2018
| @@ -106,9 +126,11 @@ func New( | |||
| // is closed, at which point it will shutdown the workqueue and wait for | |||
| // workers to finish processing their current work items. | |||
| func (c *TrainingJobController) Run(threadiness int, maxLoadDesired float64, stopCh <-chan struct{}) error { | |||
| // TODO add a lock to ensure there is only one controller in the cluster | |||
There was a problem hiding this comment.
resource lock has been implemented, so the TODO is needless.
m3ngyang
commented
Jul 12, 2018
| @@ -125,9 +147,12 @@ func (c *TrainingJobController) Run(threadiness int, maxLoadDesired float64, sto | |||
| go wait.Until(c.runWorker, time.Second, stopCh) | |||
| } | |||
|
|
|||
| // gc := NewGarbageCollector(c.KubeCli, c.trainingjobLister) | |||
| // go gc.CleanOrphans(10 * time.Minute) | |||
There was a problem hiding this comment.
delete commits and add a TODO for gc
typhoonzero
reviewed
Jul 20, 2018
| @@ -34,6 +34,7 @@ var ( | |||
| func main() { | |||
| masterURL := flag.String("master", "", "Address of a kube master.") | |||
| kubeConfig := flag.String("kubeconfig", "", "Path to a kube config. Only required if out-of-cluster.") | |||
| autoClean := flag.Bool("autoclean", false, "Auto clean pods after terminating job, default false") | |||
There was a problem hiding this comment.
So make this default false means user may need to get the logs?
There was a problem hiding this comment.
Yes, if flag autoclean is false, controller will maintain pods after success or failure. Otherwise, all pods will be deleted automatically. It's useful to debug and get logs.
tizhou86
approved these changes
Sep 18, 2018
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
fix #26