[BUG]: MSBuild Task stopped finding MSBuild.exe file correctly

[Ligtorn]

New issue checklist

• I searched for existing GitHub issues
• I read pipeline troubleshooting guide
• I checked how to collect logs

Task name

MSBuild

Task version

All

Issue Description

I specified all task versions, as the problem is a combination of version used in assembly Microsoft.Build.Utilities.Core.dll and the powershell task

I have self hosted agent version 3.248.0 on my Azure DevOps account. it is hosted on a recently installed Win11 and I just have Visual Studio 2022 Enterprise Preview 17.13.0 Preview 2.0 and Visual Studio 2022 community edition Version 17.12.3 (Non Preview) installed. When I run a build against that build agent, it doesn’t pick up any of those editions, even though it detects them fine in the beginning, if you run it with debug=true

"C:\Agent\_work\_tasks\MSBuild_c6c4c611-aa2e-4a33-b606-5eaba2196824\1.246.3\node_modules\azure-pipelines-tasks-msbuildhelpers\tools\vswhere.exe" -version [17.0,18.0) -latest -format json
##[debug][
##[debug]  {
##[debug]    "installationName": "VisualStudioPreview/17.13.0-pre.2.0+35605.110",
##[debug]    "installationPath": "C:\\Program Files\\Microsoft Visual Studio\\2022\\Preview",
##[debug]    "installationVersion": "17.13.35605.110",
##[debug]    "displayName": "Visual Studio Enterprise 2022",
##[debug]    "updateDate": "2024-12-10T18:54:55.7707095Z",
##[debug]    "enginePath": "C:\\Program Files (x86)\\Microsoft Visual Studio\\Installer\\resources\\app\\ServiceHub\\Services\\Microsoft.VisualStudio.Setup.Service",
##[debug]    "resolvedInstallationPath": "C:\\Program Files\\Microsoft Visual Studio\\2022\\Preview",
##[debug]  }
##[debug]]

I have debugged the powershell task and found out that the issue comes from \MSBuild_c6c4c611-aa2e-4a33-b606-5eaba2196824\1.246.3\node_modules\azure-pipelines-tasks-msbuildhelpers\PathFunctions.ps1, it works if the script is execute from powershell core, but not when it is executed from powershell 5.1, which is that the task run is doing.

I did a bit more debugging and found out that this scripts dynamically loads the assembly “MSBuild\Current\Bin\Microsoft.Build.Utilities.Core.dll” from my Visual Studio 2022 Enterprise Preview 17.13.0 installation folder, and it calls the Microsoft.Build.Utilities.ToolLocationHelper method for finding the MSBuild.exe file for 64 bit edition. However inside Microsoft.Build.Utilities.Core.dll assembly recent changes have been done, so that will use a new assembly called Microsoft.IO.Redist for doing a file.exists for the found msbuild.exe path. That method returns false, even when the path exists. it happens, because that assembly now requires System.Memory loaded and that doesn’t happen, hence it gets a FileLoadException exception, which gets swallowed and when it just returns false. This false value means that the agent code doesn’t think that the msbuild.exe path exists and attempts to find alternatives. This is better explained in this post dotnet/msbuild#9223 (comment)

This means as more and more agents are upgraded to newer visual studios editions with new dependency for Microsoft.IO.Redist, it would lead to more and more issues and resolving to lower editions.

I suspect that the same issue is present in vsbuild task, but I haven’t tried it out

I don’t know if the agent tasks, needs some assembly redirects or manual load of System.Memory, or if the issue should be fixed in Microsoft.IO.Redist, so I will let the experts look into that

I have made a simple powershell script which can show the issue, depending if you are running it in Windows Powershell or Powershell core. I am guessing that it works in Powershell Core, because System.Memory is already loaded

msbuildbug.ps1.txt

Environment type (Please select at least one enviroment where you face this issue)

• Self-Hosted
• Microsoft Hosted
• VMSS Pool
• Container

Azure DevOps Server type

dev.azure.com (formerly visualstudio.com)

Azure DevOps Server Version (if applicable)

No response

Operation system

Windows 11

Relevant log output

No error message, as it doesn't fail, just fallback to a very old MSBuild version 4.0, which is to old to build sdk style C# projects, so it fails in the build step.

"C:\Windows\Microsoft.NET\Framework64\v4.0.30319\msbuild.exe" "C:\agents\_work\1\s\Source\Soendergaard.SqlValidator.sln" /nologo /nr:false  /p:platform="Any CPU" /p:configuration="Release"
  Building solution configuration "Release|Any CPU".
##[debug]Processed: ##vso[task.logdetail parentid=;name=Soendergaard.SqlValidator.App\Soendergaard.SqlValidator.App.csproj.metaproj;type=Build]
##[warning]C:\agents\_work\1\s\Source\Soendergaard.SqlValidator.App\Soendergaard.SqlValidator.App.csproj.metaproj(0,0): Warning MSB4078: The project file "Soendergaard.SqlValidator.App\Soendergaard.SqlValidator.App.csproj" is not supported by MSBuild and cannot be built.

Full task logs with system.debug enabled

 [REPLACE THIS WITH YOUR INFORMATION] 

Repro steps

Run the attached powershell script in windows powershell, on a computer with the latest visual studio 2022 and no other visual studio or build tools. it should pick up the msbuild.exe path, but it doesn't, however it does if it is running under powershell.core

[praval-microsoft]

Please attach "Full task logs with system.debug enabled"

[Ligtorn]

Here is the log files and a test project. The logs doesn't show much, just that it didn't find the MSBuild.exe path from the VS2022, so it continues and fall back to a very old version

logs_335.zip
MSBuildBug.zip

[Ligtorn]

I recreated a new agent with the latest VS 2022 community edition and it worked, this left me puzzled, so I installed the latest VS2022 community preview edition and then it failed again, so it only happens when the latest preview is installed and higher that the non preview, because the vswhere still finds the the Microsoft.Build.Utilities.Core from the preview folder and that gets dynamically loaded. The preview version 17.13.0.56805 and the non preview is version 17.12.12.57101

Basically it is an new issue only in preview edition, as it upgraded to a newer version Microsoft.Build.Utilities.Core.

you can try it out by installing only preview edition on a clean agent or use my powershell script and redirect to load the preview version of Microsoft.Build.Utilities.Core

[Ligtorn]

@praval-microsoft

Thanks for looking into this issue, however I don’t agree that it is MSBuild.exe team issue. This is a issue with MSBuild/VSbuild azure develop task NOT the msbuild.exe or visual studio product, as both of them works fine with the latest changes, because they properly handle the new version dependency for system.memory. This is only a problem with MSBuild task, as they are piggybacking on the use the other teams assembly directly, without coping with the new dependency. E.g. if you used Newtonsoft assembly and that started to use a new assembly, you as the user of that assembly needs to cope with that change. Don’t expect Newtonsoft team to fix your issues

It seems that the MSBuild.exe team implemented support for long file paths via the use of Microsoft.IO.Redist.dll, as System.IO.File.Exists doesn’t support long file names, this introduced newer dependencies to assemblies, which they are entitled too, as long as they didn’t break msbuild.exe or visual studio

This issue is a time bomb waiting to happen, because as soon as the latest Visual Studio Preview becomes a production version, and the customers begin to install it on agents, there will be a huge increase in build issues, which they would have a hard time to find out why it happen and how to get around it.

We have small time window, where a solution could be implemented, before Visual Studio Preview becomes a production version, so the MSBuild/VSBuild azure devops task teams should look into this and fix it by one of these solutions

• Manually load the new dependencies in _work_tasks\MSBuild_c6c4c611-aa2e-4a33-b606-5eaba2196824\1.246.3\node_modules\azure-pipelines-tasks-msbuildhelpers\PathFunctions.ps1
• Use assembly redirects to cope with the newer version requirement for System.Memory and others
• Upgrade the task to use powershell.core instead of powershell for windows, as they are no issues in powershell.core

[praval-microsoft]

Hey, thanks for the details. I am looking into it. I tried reproducing the issue with the script that you have provided. Powershell core works fine, however Powershell 5 didn't.

Let me dig more into it and I'll update the thread

[praval-microsoft]

@Ligtorn Did adding System.Memory.dll make the above script you provided worked? I tried adding the assembly but it didn't worked for me.

[Ligtorn]

@praval-microsoft no, I never manage to get the problem solved in my powershell script, otherwise there would have been a Pull Request. This area is a bit over my head regarding to reflection usage with signed assemblies

To find the real error, I used this test c# program MSBuild.Framework.zip, which is the same code as the powershell task, where I enabled break on "System.IO.FileNotFoundException: " and there I have made a debug version of the used assemblies from this git repo https://github.com/dotnet/msbuild

This is the exception which gets thrown and ignored

System.IO.FileLoadException: 'Could not load file or assembly 'System.Memory, Version=4.0.1.1, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)'

BTW if the same assembly is statically referenced without using assembly.load, it works

	internal class Program
	{
		static void Main(string[] args)
		{
			string msBuildPath = Microsoft.Build.Utilities.ToolLocationHelper.GetPathToBuildToolsFile("MSBuild.exe", "17.0", DotNetFrameworkArchitecture.Bitness64);
			Console.WriteLine($"msBuildPath = {msBuildPath}");

		}
	}