Using Azure Test Plans Associated Automation

[ReneSchumacher]

Original comment from Create Work Item Q&A in VS Marketplace:

I'm very disappointed in this comment from Microsoft on 2022-01-28 - "Tbh, associating automated tests with test cases in Azure Test Plans is something I've been trying to move people away from for a couple years now. There's no huge benefit in doing so, while it makes things far more complicated and Azure Test Plans is an area we're not really investing in anymore."

I see a huge benefit in linking test cases to automations, as the status of your test cases is automatically updated in your test plan after the test run, and you have the ability to run a test directly from the plan. The latter gives a BA the ability to run a test without having to setup the project locally.

Is there a better option offered by MS?

[ReneSchumacher]

I would not discourage people from using Azure Test Plans for the purpose for which it was designed. This was and is managing manual test efforts. Thus, if you are managing and running manual tests, Azure Test Plans is the tool to use.

The ability to associate automation to test cases was created with a very special intent: we wanted customers to start with a manual test during which testers would record their actions, so that developers could then create Coded UI Tests from these recordings and link them back to the test case to run them automatically by running an automated test step for the test plan containing the tests.

However, this has never worked well enough, so most customers didn't invest too much effort into this. Consequently, we deprecated Coded UI Tests, removed them from Visual Studio, and told customers to move to other UI test frameworks/tools like Selenium or Appium. In addition, the setup for being able to manually trigger an automated test run from Azure Test Plans is rather complicated (it requires a correctly configured build pipeline and a correctly configured classic release), and I have never really seen this being used by customers on a larger scale.

As mentioned in my original comment, there is not a very large benefit of running tests that are associated with test plan compared to just running the automated tests as part of a pipeline and looking at the test results there. It's true that it results in a central place for all test results. However, my recommendation would always be to keep test results close to the system under test. Thus, if you test a shared component, I wouldn't recommend creating a test plan for that, putting "empty" test cases in there, and linking all your automated tests to them. This is a lot of overhead and requires every developer to remember to create this artificial test case whenever a new test has been written. I believe that it is far better to keep the tests in the pipeline for the shared component (service, application, etc.) and track the quality of your components by looking at the pipeline status. If details about the different tests are needed, it is still possible to look into the test run of that pipeline.

In a complex system where someone wants to see the aggregated test results from all components of the system, I would rather rely on an external report that pulls together all the necessary pipeline data than relying on the discipline of every developer to create tests cases along with test automation.

I'm happy to discuss this further 🙂