QAP Test Book

The following wiki page contains ideas for tests to perform on new releases of QAP.

Installation Tests

• Does QAP install all of its dependencies?

Config File Tests

• num_subjects_per_bundle
• Input validation: nonzero positive int (valid), zero (invalid), negative int (invalid), float (invalid), string (invalid)
• Set to a value greater than the number of subjects in the subject list.
• num_bundles_at_once
  • Input validation: nonzero positive int (valid), zero (invalid), negative int (invalid), float (invalid), string (invalid)
  • Set to a value greater than the number of subjects in the subject list.
• resource_manager
• Check that it uses correct resource manager.
• Check that it returns an error and won't continue for invalid resource managers.
• output_directory
• Set to path that contains spaces.
• working_directory
• Set to path that contains spaces.
• write_all_outputs
• Input validation: boolean (valid), non-boolean (invalid), integer 0 or 1 (valid)
• Check if it behaves as expected (not saving outputs when set to False).
• Check if it interacts with other boolean values (upload_to_s3).
• upload_to_s3
• Input validation: boolean (valid), non-boolean (invalid), integer 0 or 1 (valid)
• bucket_prefix
• Point to a non-existent bucket prefix (expected behavior: throws an error).
• Point to an unreadable bucket prefix (expected behavior: throws an error).
• Use spaces in the path (expected behavior: throws an error).
• bucket_out_prefix
• Point to a non-existent bucket prefix (expected behavior: throws an error).
• Point to an unwritable bucket prefix (expected behavior: throws an error).
• Use spaces in the path (expected behavior: throws an error).
• local_prefix
• Point to a non-existent path (expected behavior: throws an error).
• Point to an unwritable path (expected behavior: throws an error).
• Use spaces in the path (expected behavior: throws an error).
• Try to download more data than there is space for (expected behavior: throws a warning).
• bucket_name
• Input validation: Make sure that all input is cast into strings (i.e., try to see if '5' or 'True' are still read in / used as integers).
• Point to a nonexistent bucket (expected behavior: throws an error).
• Point to a bucket that you don't have the right to access (expected behavior: throws an error).
• creds_path
• Point to a nonexistent CSV (expected behavior: throws an error).
• Point to a csv that does not follow the AWS credential format (expected behavior: throws an error).
• In general:
• Try to mix and match new YAML keys with keys from previous versions of QAP (expected behavior: deprecated keys should be ignored).

Benchmarking

• Get timing for different permutations of several cores for a 2 subject run (combine w/ num_subjects_per_bundle and num_subjects_per_run tests perhaps).