Tasks integration: Failing tests?

[The-Compiler]

Tests seem to fail after merging the tasks integration: https://github.com/The-Compiler/vscode-python-tox/runs/7218607915?check_suite_focus=true

  1) ToxTaskProvider Test Suite
       getting tox tasks:
AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:
0 !== 3
+ expected- actual
-0
+3

not sure what's going on there? I saw this error once locally, but when rerunning it went away, so I didn't think much of it...

@dariuszparys do you have any idea what could be going on there?

(follow-up to #5/#12)

[The-Compiler]

After 2 hours of debugging I think I finally got to the bottom of this and figured out how to make those tests run 😅

So the problem is that it's not possible to enter a new workspace in tests, because VS Code restarts the extension host when that's done, and thus outright refuses to do so when running tests:

• WorkspaceFolders can't be changed while testing microsoft/vscode#78740
• Investigation: do not restart Extension host when first folder changes microsoft/vscode#69335

When running the tests for the first time, that's confirmed by an error:

Extension 'python-tox' failed to update workspace folders: Error: Entering a new workspace is not possible in tests.

Triggered from here:

vscode-python-tox/src/test/suite/toxTaskProvider.test.ts

Line 22 in 3bc0159

vscode.workspace.updateWorkspaceFolders(0, null, allEnvsWorkspaceFolder);

And indeed the tests fail because the task runner can't find any workspace - vscode.workspace.workspaceFolders is empty here:

vscode-python-tox/src/toxTaskProvider.ts

Lines 78 to 80 in 46ecf1f

	if (!workspaceFolders || workspaceFolders.length === 0) {
	return result;
	}

However, the workspace state seems to get stored in .vscode-test/ despite the error! Thus, when launching the tests a second time, they just happen to open allenvs as a workspace and everything works, until .vscode-test/ is deleted again... Of course, the CI always starts with a clean state, which is why they probably always passed for @dariuszparys locally, but always fail on the CI. To top it all off, apparently I never ever enabled CI for PRs and didn't even notice until now... 🤭

In 46ecf1f, I now ended up changing runTest.js to open all examples as folders in a multi-folder workspace - with that setup, everything works again. Not perfect since it means tests aren't isolated as nicely from each other anymore, but better than nothing I guess...

[dariuszparys]

Thanks @The-Compiler for fixing, totally oversaw this, I'm just curious why my merge was successful with the CI in place, I assume that the CI triggered before the merge can happen should have caught that, right?

[The-Compiler]

Indeed it should have - but the CI actually didn't run on pull requests at all, because I had an indentation error in the job definition 😊 f7791a2