Add "describe" strategy to skip only tests inside same describe

[quad5]

Describe the bug
A clear and concise description of what the bug is.

This is probably related to #100. This ticket is to report an issue where failFast behavior from previous describe is affecting next describe. See image. Notice that when a test fails in previous failfast, it is skipping all the tests in the next describe.

To Reproduce
Steps to reproduce the behavior

Expected behavior
A clear and concise description of what you expected to happen.

Logs
If applicable, add logs to help explain your problem.

describe('in failFast1, describe 1', { "failFast": {"enabled": true} }, function() { it('failFast1', function() { cy.log('testing in failFast1 file'); expect(1).equal(0); }) }) describe('in failFast1, describe 2', { "failFast": {"enabled": true} }, function() { it('failFast1', function() { cy.log('testing in failFast1 file'); expect(6).equal(0); }) })

** Operating system, Node.js an npm versions, or browser version (please complete the following information):**

OS: [e.g. Ubuntu 18.04] - Mac
Node.js: [e.g. 8.11.1] - v12.18.4
npm: [e.g. 5.6.0] - 6.14.10
Browser: [e.g. Chrome 73.0.3683] - Electron 87
failFast: 2.3.2

Additional context
Add any other context about the problem here.

[javierbrea]

Hi @quad5,

You're right, this issue is related to #100. As commented there: "This is the expected behaviour. The failFast.enabled option produces enabling the skip flag, which produces skipping all of the rest of tests. It can be understood as "Skip the rest of tests if any test inside this block fails", not as "Skip the rest of tests inside this block if any test inside this block fails". It does not provide a "scope" for the fail-fast feature, it provides a mechanism to decide what tests should enable the skip flag or not."

In this case, there is no mechanism to avoid that one block of tests inside the same spec file is affected by the "skip" flag, because that behaviour is related to the FAIL_FAST_STRATEGY environment variable, and it only supports "spec" and "run" options for the moment. So, the only alternative would be to separate the blocks in different spec files and set CYPRESS_FAIL_FAST_STRATEGY=spec

For the moment, the plugin relies on the Cypress.runner.stop method internally, and this does not allow to decide which tests should be executed or not, it simply skips all of the rest of tests in the current spec. This produces some issues, as those described in #88 and #91. I have to investigate further in order to find an alternative implementation (ideally, one that would not require to abuse of Cypress private methods or properties). If I find it, maybe a new value "describe" could be supported in the FAIL_FAST_STRATEGY option, which would produce to skip tests only in the current describe.

Thanks for your feedback! 🙂

[alecmestroni]

Any news on this improvement?