Maximum execution time is misleading as it splits time equally between all tests

[ArmorKnightPlus]

When looking at how VPL runs tests, it appears that it treats its Maximum Execution Time as a resource to be equally split among all tests:

int timeout = maxtime / testCases.size();

When setting the Maximum Execution Time on a VPL activity, this is not explained anywhere. This gives the impression that there is a single time limit that, when elapsed, causes any remaining tests to fail.

This seems like an odd choice; not all tests are created equal! What if an assignment is made with 4 tests and a time limit of 4 minutes; instead of giving each test an arbitrary time limit of 1 minute, why is the time not used for the entire group? For example, as VPL exists today, a Maximum Execution Time of 4 minutes would not allow a VPL activity with 4 tests to run one of the tests for 3 minutes, and the remaining tests for 1 minute. Instead, the time is rigidly and automatically divided equally between all tests.

To be clear, I don't think that means each test needs its own unique time limit; rather it makes more sense to report a global timeout and the amount of time spent per test, rather than dividing the time equally among all tests.

[jcrodriguez-dis]

Dear @ArmorKnightPlus,

Thank you for pointing out this issue, which has been a known limitation for some time. The good news is that the next release of VPL will address this with a new version of the default VPL evaluator. This updated evaluator introduces enhanced customization options for the report generation and includes improved time management features.
Specifically, you will be able to set:

1. A global time limit for the evaluation process, which will act as an primary timeout.
2. Time limits for specific test cases, allowing for greater flexibility when certain tests require more execution time than others.

We believe these enhancements will fulfill your request by providing the flexibility to allocate time more effectively without rigidly dividing it equally among all tests. This will allow for better handling of scenarios where different tests require different execution times, as you've described.

Note: The existing behavior of splitting the global time limit among all test cases has been retained for compatibility, ensuring that older configurations will continue to function as expected. Setting a specific time limit for each test case ensures that a single poorly performing test will not prevent the remaining tests from being executed.

Thank you again for your valuable feedback, and we hope the upcoming features will meet your expectations.

Best regards,
Juan Carlos