Skip to content

Commit

Permalink
Add explanation about imports and global variables
Browse files Browse the repository at this point in the history
  • Loading branch information
WardLT committed Oct 17, 2023
1 parent 2dee78a commit bbcd64d
Show file tree
Hide file tree
Showing 4 changed files with 122 additions and 1 deletion.
3 changes: 3 additions & 0 deletions docs/teaching_scripts/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# Example Scripts

Scripts which illustrate example from the documentation that do not run well as part of the pytest
60 changes: 60 additions & 0 deletions docs/teaching_scripts/test_apps.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
"""Tests documentation related to building apps. Must reside outside the Parsl library to be effective"""
from typing import List, Union

import numpy as np

from parsl import python_app, HighThroughputExecutor, Config
import parsl

parsl.load(Config(executors=[HighThroughputExecutor(label='htex_spawn', max_workers=1, start_method='spawn', address='127.0.0.1')]))


# Part 1: Explain imports
# BAD: Assumes library has been imported
@python_app(executors=['htex_spawn'])
def bad_imports(x: Union[List[float], np.ndarray], m: float, b: float):
return np.multiply(x, m) + b


# GOOD: Imports libraries itself
@python_app(executors=['htex_spawn'])
def good_imports(x: Union[List[float], 'np.ndarray'], m: float, b: float):
import numpy as np
return np.multiply(x, m) + b


future = bad_imports([1.], 1, 0)

try:
future.result()
raise ValueError()
except NameError as e:
print('Failed, as expected. Error:', e)

future = good_imports([1.], 1, 0)
print(f'Passed, as expected: {future.result()}')

# Part 2: Test other types of globals
# BAD: Uses global variables
global_var = {'a': 0}


@python_app
def bad_global(string: str, character: str = 'a'):
global_var[character] += string.count(character) # `global_var` will not be accessible


# GOOD
@python_app
def good_global(string: str, character: str = 'a'):
return {character: string.count(character)}


try:
bad_global('parsl').result()
except NameError as e:
print(f'Failed, as expected: {e}')

for ch, co in good_global('parsl', 'a').result().items():
global_var[ch] += co

58 changes: 58 additions & 0 deletions docs/userguide/apps.rst
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,64 @@ as in following code snippet, which copies the contents of one file (``in.txt``)
echo(inputs=[in.txt], outputs=[out.txt])
Imports and Global Variables
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Parsl apps have access to less information from the script that defined then
than functions run via Python's native multiprocessing libraries.
The reasons is that functions are executed on workers that
do not have access to the same global variables as the script that defined them.
Practically, this means

1. *Functions may need to re-import libraries.*
Place the import statements that define functions or classes inside the function.
Type annotations should also not use libraries that must be defined later.

.. code-block:: python
import numpy as np
# BAD: Assumes library has been imported
@python_app
def linear_model(x: list[float] | np.ndarray, m: float, b: float):
return np.multiply(x, m) + b
# GOOD: Function imports libraries on remote worker
@python_app
def linear_model(x: list[float] | 'np.ndarray', m: float, b: float):
import numpy as np
return np.multiply(x, m) + b
2. *Global variables are inaccessible*.
Functions should not use variables defined outside the function.
Likewise, do not assume that variables created inside the function are visible elsewhere.


.. code-block:: python
# BAD: Uses global variables
global_var = {'a': 0}
@python_app
def counter_func(string: str, character: str = 'a'):
global_var[character] += string.count(character) # `global_var` will not be accessible
# GOOD
@python_app
def counter_func(string: str, character: str = 'a'):
return {'A'} string.count(character) # `global_var` will not be accessible
for ch, co in counter_func('parsl', 'a').result()
global_var[ch] += co
.. note::

These rules do not apply to functions which are imported from libraries.
Library functions are sent to workers differently than functions defined in a script.

Special Keyword Arguments
^^^^^^^^^^^^^^^^^^^^^^^^^^

Expand Down
2 changes: 1 addition & 1 deletion docs/userguide/workflow.rst
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ Sequential workflows can be created by passing an AppFuture from one task to ano
def generate(limit):
from random import randint
"""Generate a random integer and return it"""
return randint(1,limit)
return randint(1, limit)
# Write a message to a file
@bash_app
Expand Down

0 comments on commit bbcd64d

Please sign in to comment.