Added the ability to re-run failed tests and merge the output files post re-run #35
Conversation
run_robot_test.sh
Outdated
| @@ -1,5 +1,5 @@ | |||
| #!/usr/bin/env bash | |||
| set -eux | |||
| set -x | |||
There was a problem hiding this comment.
Is this intentional change? IMHO "-eu" wont harm anything
There was a problem hiding this comment.
Yes, they both had to be removed due to requirement changes
-e was removed because it causes the shell script to exit when there is robot test failure, as the exit status of robot command is 1
So if a test case fails, the bash script will exit right away and won't execute the code for re-run failed tests
-u was removed as it requires that every variable used should already be set, and we are using a Jenkins Environment Variable which outside of Jenkins run will be empty/un-set
run_robot_test.sh
Outdated
| ;; | ||
| Linux) | ||
| TEST_ARTIFACT_DIR=$(mktemp -d -p ${TEST_ARTIFACT_DIR} -t rhoda-ci-$(date +%Y-%m-%d-%H-%M)-XXXXXXXXXX) | ||
| TEST_ARTIFACT_DIR="${TEST_ARTIFACT_DIR}/rhoda-${BUILD_NUMBER}-$(date +%Y-%m-%d-%M%S)" |
There was a problem hiding this comment.
${BUILD_NUMBER} I dont see the declaration for this variable, is this expected to be passed as CLI argument?
There was a problem hiding this comment.
${BUILD_NUMBER} is a Jenkins Environment variable, it will automatically get picked up once a Job is kicked off
https://wiki.jenkins-ci.org/display/JENKINS/Building+a+software+project#Buildingasoftwareproject-belowJenkinsSetEnvironmentVariables
When running manually, the variable will be empty, so won't be included in the directory name
There was a problem hiding this comment.
I am not sure Jenkins environment variables can be called this way, I use ${env.BUILD_NUMBER} to access the variable.
Can you please attach the test results from local jenkins setup?
There was a problem hiding this comment.
I haven't run this from local jenkins setup
${BUILD_NUMBER} should also work the same theoretically - https://www.lambdatest.com/blog/set-jenkins-pipeline-environment-variables-list/#Reading
I can change it to ${env.BUILD_NUMBER} if it makes more sense, please suggest
There was a problem hiding this comment.
I am not sure too. So please setup the local jenkins and give it a try
run_robot_test.sh
Outdated
| @@ -168,6 +168,13 @@ case "$(uname -s)" in | |||
| esac | |||
|
|
|||
|
|
|||
| ./venv/bin/robot -d "${TEST_ARTIFACT_DIR}" -x xunit_test_result.xml -r test_report.html ${TEST_VARIABLES} --variablefile ${TEST_VARIABLES_FILE} ${EXTRA_ROBOT_ARGS} ${TEST_CASE_FILE} | |||
| ./venv/bin/robot ${EXTRA_ROBOT_ARGS} -d ${TEST_ARTIFACT_DIR} -x xunit_test_result.xml -r test_report.html ${TEST_VARIABLES} --variablefile "${TEST_VARIABLES_FILE}" "${TEST_CASE_FILE}" | |||
There was a problem hiding this comment.
On the variable expansion, shall we follow same format everywhere without using double quotes around the variable? LMKWYT
There was a problem hiding this comment.
I agree, the standard and best way to use variables is inside double quotes to avoid incorrect expansion. In our case though, problem is that it can only be used for non-empty variables
If empty string variables are put inside double quote, robot tries to interpret them as command line input and fails to run.
I should, however, add the double quotes for ${TEST_ARTIFACT_DIR}
Thank you!
| if [[ $? -ne 0 ]] && [[ -f "${TEST_ARTIFACT_DIR}/output.xml" ]]; then | ||
| ./venv/bin/robot ${EXTRA_ROBOT_ARGS} -d ${TEST_ARTIFACT_DIR} -x xunit_test_result.xml -r test_report.html ${TEST_VARIABLES} --variablefile "${TEST_VARIABLES_FILE}" --rerunfailed "${TEST_ARTIFACT_DIR}/output.xml" --output "output_rerun.xml" "${TEST_CASE_FILE}" | ||
|
|
||
| ./venv/bin/rebot -d ${TEST_ARTIFACT_DIR} -o "output.xml" -x xunit_test_result.xml -r test_report.html --merge "${TEST_ARTIFACT_DIR}/output.xml" "${TEST_ARTIFACT_DIR}/output_rerun.xml" |
There was a problem hiding this comment.
Please attach the test results for the successful execution of rerun passed cases
There was a problem hiding this comment.
The logs are already attached, did I miss something in there?
There was a problem hiding this comment.
I am good with logs. I was looking for report, xunit and output xml files
There was a problem hiding this comment.
Thanks Rajan, attached the complete results directory
harshkumarRH
force-pushed
the
run-script
branch
2 times, most recently
from
5a5e1ed to
8f83c7a
Compare
…ost re-run Signed-off-by: Harsh Kumar <[email protected]>
|
"This pull request now has conflicts with the target branch. |
|
This pull request now has conflicts with the target branch. Please resolve conflicts. |
1 similar comment
|
This pull request now has conflicts with the target branch. Please resolve conflicts. |
With this PR, the below changes were made to run_robot_test.sh
Marks tentative completion of DBAAS-544
Logs for reference: PR35.zip
Complete Test Result directory: rhoda--2022-06-13-3249.zip
Signed-off-by: Harsh Kumar [email protected]