run par as an entrypoint if there is no patch or jetter patch. #994
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
facebook-github-bot
added
the
CLA Signed
|
This pull request was exported from Phabricator. Differential Revision: D66621649 |
|
This pull request was exported from Phabricator. Differential Revision: D66621649 |
yikaiMeta
added a commit
to yikaiMeta/torchx
that referenced
this pull request
Dec 17, 2024
…ch#994) Summary: Pull Request resolved: pytorch#994 # Context: Currently, when running torchx local job, we are using penv_python as entrypoint. That means we pass the actual .par or .xar file as argument to penv_python. within penv_python, the par/xar is executed as a new process. # Old way to run torchx local job. For example, if the local job is running "jetter --help", torchx runs it like: PENV_PAR='/data/users/yikai/fbsource/buck-out/v2/gen/fbcode/a6cb9616985b22b0/jetter/__jetter-bin__/jetter-bin-inplace.par' penv_python -m jetter.main --help It passes the par file as an environment variable called "PENV_PAR"(There is another way to pass this to penv_python, which is passing 'PENV_PARNAME' as env variable then get the par file's path using it. But it is very very rare, only 0.1% of total usage.) # New way to run torchx local job After migration, We will run it like: PAR_MAIN_OVERRIDE=jetter.main /data/users/yikai/fbsource/buck-out/v2/gen/fbcode/a6cb9616985b22b0/jetter/__jetter-bin__/jetter-bin-inplace.par --help NOTE: This diff only migrates one of the most common use cases, which: 1. There are no patch or jetter patch. 2. it's a par not xar. 3. the par file is passed via "PENV_PAR" env variable. For other use cases, we still run penv_python as entrypoint. Reviewed By: Sanjay-Ganeshan Differential Revision: D66621649
yikaiMeta
force-pushed
the
export-D66621649
branch
from
a4eeb3c to
701e972
Compare
|
This pull request was exported from Phabricator. Differential Revision: D66621649 |
yikaiMeta
added a commit
to yikaiMeta/torchx
that referenced
this pull request
Dec 17, 2024
…ch#994) Summary: Pull Request resolved: pytorch#994 # Context: Currently, when running torchx local job, we are using penv_python as entrypoint. That means we pass the actual .par or .xar file as argument to penv_python. within penv_python, the par/xar is executed as a new process. # Old way to run torchx local job. For example, if the local job is running "jetter --help", torchx runs it like: PENV_PAR='/data/users/yikai/fbsource/buck-out/v2/gen/fbcode/a6cb9616985b22b0/jetter/__jetter-bin__/jetter-bin-inplace.par' penv_python -m jetter.main --help It passes the par file as an environment variable called "PENV_PAR"(There is another way to pass this to penv_python, which is passing 'PENV_PARNAME' as env variable then get the par file's path using it. But it is very very rare, only 0.1% of total usage.) # New way to run torchx local job After migration, We will run it like: PAR_MAIN_OVERRIDE=jetter.main /data/users/yikai/fbsource/buck-out/v2/gen/fbcode/a6cb9616985b22b0/jetter/__jetter-bin__/jetter-bin-inplace.par --help NOTE: This diff only migrates one of the most common use cases, which: 1. There are no patch or jetter patch. 2. it's a par not xar. 3. the par file is passed via "PENV_PAR" env variable. For other use cases, we still run penv_python as entrypoint. Reviewed By: Sanjay-Ganeshan Differential Revision: D66621649
yikaiMeta
force-pushed
the
export-D66621649
branch
from
701e972 to
0776559
Compare
yikaiMeta
added a commit
to yikaiMeta/torchx
that referenced
this pull request
Dec 18, 2024
…ch#994) Summary: # Context: Currently, when running torchx local job, we are using penv_python as entrypoint. That means we pass the actual .par or .xar file as argument to penv_python. within penv_python, the par/xar is executed as a new process. # Old way to run torchx local job. For example, if the local job is running "jetter --help", torchx runs it like: PENV_PAR='/data/users/yikai/fbsource/buck-out/v2/gen/fbcode/a6cb9616985b22b0/jetter/__jetter-bin__/jetter-bin-inplace.par' penv_python -m jetter.main --help It passes the par file as an environment variable called "PENV_PAR"(There is another way to pass this to penv_python, which is passing 'PENV_PARNAME' as env variable then get the par file's path using it. But it is very very rare, only 0.1% of total usage.) # New way to run torchx local job After migration, We will run it like: PAR_MAIN_OVERRIDE=jetter.main /data/users/yikai/fbsource/buck-out/v2/gen/fbcode/a6cb9616985b22b0/jetter/__jetter-bin__/jetter-bin-inplace.par --help NOTE: This diff only migrates one of the most common use cases, which: 1. There are no patch or jetter patch. 2. it's a par not xar. 3. the par file is passed via "PENV_PAR" env variable. For other use cases, we still run penv_python as entrypoint. Reviewed By: Sanjay-Ganeshan Differential Revision: D66621649
yikaiMeta
force-pushed
the
export-D66621649
branch
from
0776559 to
8a73707
Compare
|
This pull request was exported from Phabricator. Differential Revision: D66621649 |
…ch#994) Summary: # Context: Currently, when running torchx local job, we are using penv_python as entrypoint. That means we pass the actual .par or .xar file as argument to penv_python. within penv_python, the par/xar is executed as a new process. # Old way to run torchx local job. For example, if the local job is running "jetter --help", torchx runs it like: PENV_PAR='/data/users/yikai/fbsource/buck-out/v2/gen/fbcode/a6cb9616985b22b0/jetter/__jetter-bin__/jetter-bin-inplace.par' penv_python -m jetter.main --help It passes the par file as an environment variable called "PENV_PAR"(There is another way to pass this to penv_python, which is passing 'PENV_PARNAME' as env variable then get the par file's path using it. But it is very very rare, only 0.1% of total usage.) # New way to run torchx local job After migration, We will run it like: PAR_MAIN_OVERRIDE=jetter.main /data/users/yikai/fbsource/buck-out/v2/gen/fbcode/a6cb9616985b22b0/jetter/__jetter-bin__/jetter-bin-inplace.par --help NOTE: This diff only migrates one of the most common use cases, which: 1. There are no patch or jetter patch. 2. it's a par not xar. 3. the par file is passed via "PENV_PAR" env variable. For other use cases, we still run penv_python as entrypoint. Reviewed By: Sanjay-Ganeshan Differential Revision: D66621649
yikaiMeta
force-pushed
the
export-D66621649
branch
from
8a73707 to
81b78b8
Compare
|
This pull request was exported from Phabricator. Differential Revision: D66621649 |
tonykao8080
approved these changes
Dec 18, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
Context:
Currently, when running torchx local job, we are using penv_python as entrypoint. That means we pass the actual .par or .xar file as argument to penv_python. within penv_python, the par/xar is executed as a new process.
Old way to run torchx local job.
For example, if the local job is running "jetter --help", torchx runs it like:
PENV_PAR='/data/users/yikai/fbsource/buck-out/v2/gen/fbcode/a6cb9616985b22b0/jetter/jetter-bin/jetter-bin-inplace.par' penv_python -m jetter.main --help
It passes the par file as an environment variable called "PENV_PAR"(There is another way to pass this to penv_python, which is passing 'PENV_PARNAME' as env variable then get the par file's path using it. But it is very very rare, only 0.1% of total usage.)
New way to run torchx local job
After migration, We will run it like:
PAR_MAIN_OVERRIDE=jetter.main /data/users/yikai/fbsource/buck-out/v2/gen/fbcode/a6cb9616985b22b0/jetter/jetter-bin/jetter-bin-inplace.par --help
NOTE: This diff only migrates one of the most common use cases, which: 1. There are no patch or jetter patch. 2. it's a par not xar. 3. the par file is passed via "PENV_PAR" env variable. For other use cases, we still run penv_python as entrypoint.
Reviewed By: Sanjay-Ganeshan
Differential Revision: D66621649