Parallel Processing Prototyping

[cczhu]

Much of TEPs-I is embarrassingly parallel, meaning it shouldn't take much effort to parallelize many jobs. Indeed, Arman does it in MATLAB with just a few flags at the start of his processes. This should be implemented in our code as well.

Technologies for parallelization:

• Multiprocessing (also (here)[https://www.machinelearningplus.com/python/parallel-processing-python/])
• joblib.parallel
• Luigi and Airflow can do batch serialization, but I'd rather not use them - way too much overhead for our purposes, and may not play well with a system-level Luigi or Airflow server running all or jobs.

One concern: *do these packages break our ability to do parallelized hyperparameter tuning with a model expermentation package?

File I/O parallelization is discussed in #6 .