Treatment of "Web.*.config" to facilitate Azure Pipeline Transformations

[leusbj]

@CZEMacLeod had mentioned wanting to start looking at a DefaultItem strategy for the various web.*.config files within a project to facilitate developers using the Azure Pipeline transformation process. So I figured I'd kick off the discussion.

The web article File transforms and variable substitution reference calls out 2 things that will need to happen to these project items to allow them to be available in the package where the pipeline can use them

• Ensure they are Content items
• Ensure they are not marked with <DependentUpon>web.config</DependentUpon>

This lines up with my understanding of the of the build/package process.

ItemType	DependentUpon Set	IncludeInPackage
None	True	Not included - By Convention "None" items are not included
None	False	Not included - By Convention "None" items are not included
Content	True	Not included - Items marked as DependentUpon the web.config are excluded from copying to the package
Content	False	Included - Is a Content Item and also not marked as DependentUpon, so not consumed as build time transform

Starting with the section that currently handles the web.*.config files that are NOT based on the build configurations

    <Content Include="Web.*.config" Exclude="@(None)">
      <DependentUpon>Web.config</DependentUpon>
    </Content>

We could remove the <DependentUpon>web.config</DependentUpon> declaration

    <Content Include="Web.*.config" Exclude="@(None)">
    </Content>

These items:

• WILL be included in the package and available for use by the pipeline
• WILL NOT appear "Nested" under the web.config (that's probably ok, because that is the behavior of legacy web application project type in VS)

Additionally, if the appearance of nesting in the IDE is important then we might consider adding a SystemWeb.DesignTime.targets with something like:

<ItemGroup 
  Condition=" '$(BuildingInsideVisualStudio)' == 'true' and '$(VisualStudioVersion)' != '' and '$(VisualStudioVersion)' >= '15.0' ">
    <ProjectCapability Include="DynamicFileNesting"/>
</ItemGroup> 

This DynamicFileNesting Capability:

• Is mentioned/included in Sdk.Web and Sdk.Razor type SDK projects import chain
• Uses a filename/expression logic to automatically nest files in the Solution Explorer even without the DependentUpon being set.
• Is supported with at least VS 2017 15.9.45 (and dotnet 2.1.202) onwards (I don't have anything older installed so I can't check, but if anyone knows more, please chime in).

I'd love to hear other ideas too!

[CZEMacLeod]

@leusbj I think most of this logic is captured in pr #44 you just submitted. Do we need to cover anything more?

I guess some of this stuff should go in the docs, although I do love the amount of comments you added to the actual src files.

I did wonder about adding custom item types for publishing profiles vs configuration transforms and using a target to put them in the correct groups for build and publish. That way they could be nested without the performance hit of dynamic file nesting.

Interestingly I had hit the problem mentioned in issue #43 myself the week before (hence having the solution immediately to hand) but hadn't had a chance to make the change to the sdk yet.

[leusbj]

@CZEMacLeod I guess I have been lucky to work with "Small Enough" projects that I had not encountered any of the performance problems. Like the <None Remove="@(CustomItemGroup)"/> idea that begins to fall down under large projects. I had read the github issue regarding that, but when I saw that it was "resolved" with one of the now-available VS2019 versions, and that some of the OOB Project SDK's use that technique, I had just assumed it was safe to do. It was your warning, that the only thing "resolved" about it was removing items from ItemGroup using a "wildcard path" ( in the last week or so, I created a dummy project with 5k controllers, 5k css files, 5k App_Code files, 5k App_Data files, and 5k razor views to help illustrate this problem to myself).

And again, I made an assumption that the DynamicFileNesting projectcapability was safe to use since it is part of every .net core Web or Razor class library project type. If it is something that we should avoid for performance, I would like to hear more about your idea.

Was the idea you thinking to do something like...

    <!-- PublishConfigTransformations are the items from which the 
               package/publish process chooses the one that 
               matches the name of the chosen build configuration
    -->
    <AvailableItemName Include"PublishConfigTransformations" />

    <!-- DeployConfigTransformations are the items from which the 
               Azure deployment pipeline chooses the one that 
               matches the name of the stage/environment being deployed to
    -->
    <AvailableItemName Include"DeployConfigTransformations" />
</ItemGroup>

Glob for them something like this (this wouldn't actually work, but something close)

    <_WebConfigConfigurations Include="$(Configurations)" />
    <PublishConfigTransformations Include="@(_WebConfigConfigurations->'Web.%(Identity).config')">
      <DependentUpon>Web.config</DependentUpon>
    </PublishConfigTransformations>

    <PublishConfigTransformations Include="Web.BindingRedirects.config" Condition="EXISTS('Web.BindingRedirects.config')">
      <DependentUpon>Web.config</DependentUpon>
    </PublishConfigTransformations>

    <DeployConfigTransformations Include="Web.*.config" Exclude="@(PublishConfigTransformations)" >
      <DependentUpon>Web.config</DependentUpon>
    </DeployConfigTransformations>

This should make them appear as nested (because of the DependentUpon property) in the Solution Explorer, and then "move" them from one ItemGroup to anther with a Target inserted somewhere before the "Package" process

<Target Name="PrepWebConfigsForPackaging BeforeTargets="WhicheverTargetIsTheRightTargetPreferrablyBeforeDeduplicationTarget" >
    <None Include="@(PublishConfigTransformations)" />
    <PublishConfigTransformations Remove="@(PublishConfigTransformations)" />

    <Content Include="@(DeployConfigTransformations )" />
    <DeployConfigTransformations Remove="@(DeployConfigTransformations )" />
</Target>

[CZEMacLeod]

I think that is roughly the direction I was thinking.
I think that maybe PublishConfigTransformations are BuildConfigTransformations, as they are applied during build and debug - there might need to be something to make them work with the xdt loopback part, but maybe it just requires that the BeforeTargets is pointing to the correct place.
The DeployConfigTransformations seems fine - although I think I would originally have called them PublishConfigTransformations, but as they are actually used during the MSDeploy phase rather than the actual publish phase your naming seems better.
I think that Web.BindingRedirects.config is just put into None directly as it is neither a deploy config, nor a build transform.
Also, I have found that if you don't have a web.xyz.config and you have a configuration xyz (even if xyz is Debug or Release), then the file is included in the tree as missing. Not the best so the BuildConfigTransformations line needs an exists check on it too.

I haven't tried any of this yet, and maybe it just isn't needed, but when I went looking for DynamicFileNesting since I wasn't aware of it, I saw lots of stuff about it being deprecated already in favour of new entries and custom file nesting rules files, along with multiple references to why it wasn't enabled in normal class libraries (e.g. performance). I think there is some discussion here dotnet/project-system#5722 (comment) and here dotnet/project-system#7161

And maybe in general use the Item Remove is fine, but again there was a lot of talk about not using that due to performance when the project sdk style was originally created and hence the use of DefaultItemExcludes and it was better to Exclude the items from the collection rather than glob them in then remove them again.
It may well be that these performance issues are now a thing of the past, but I feel better to err on the side of caution.

<_WebConfigConfigurations Include="$(Configurations)"/>
<BuildConfigTransformations Include="@(_WebConfigConfigurations->'Web.%(Identity).config')"
                            Condition="Exists('@(_WebConfigConfigurations->'Web.%(Identity).config')')">
	<DependentUpon>Web.config</DependentUpon>
	<IsTransformFile>True</IsTransformFile>
</BuildConfigTransformations>

<Target Name="PrepareWebConfigTransformations" BeforeTargets="BeforeBuild;BeforePublish" >
    <None Include="@(BuildConfigTransformations)" />
    <Content Include="@(DeployConfigTransformations )" />
</Target>

For my own education, why did you have lines clearing the PublishConfigTransformations and DeployConfigTransformations collections? I don't see any reason for that to be necessary.

[leusbj]

BuildConfiguration/PublishTransformation/AzureDevopsFileTransformation

I think I might have been missing an entire category of transformations, I read a little more after your comments, and I now realize that there are really 3 types of transformation files, that for web projects boil down to 2 "behaviors" in a web project

	BuildConfiguration Transform	PublishProfile Transform	Devops Release File Transform
How	Right-Click on web.config and choose "Add Config Transform"	Right-Click on publishprofile.pubxml and choose "Add Config Transform"	Add xml file to project
Item Metadata	None with <DependentUpon> set	None with <DependentUpon> set	Content without DependentUpon
When	Non-Web Projects -> app.config is used as the base and the transform that matches the selected build config is applied to create the /bin/MyProjectName.config. Web Projects -> the Build/Debug process runs from the web.config in the project root, and so doesn't want to overwrite a "source-controlled" file, so for web projects, it does not do transforms until the publish step	For all projects -> the transform that matches the publishprofile and the buildconfiguration is chosen and is applied to the project's config file during the package/publish phase	The Azure Pipeline step looks for a transform file that matches the name of the environment and applies that transform file to the project's config file, and the file must be present in the deployment artifact
Notes	different between web and non-web	This is one I didn't even know existed	This is the one that spawned the discussion/enhancement

I had proposed ItemGroup names as descriptions of WHEN they would be applied, but probably a poor choice on my part

• PublishConfigTransformations (because I knew that for web projects the transformations did not take place during build, but rather at "Package/Publish" time), but I like your term of BuildConfigurationTransformations better... or maybe we need both??
• DeploymentConfigTransformations (because they would used at deployment time in Azure Devops task)

You were also right to call out trying to conditionally Include the BuildConfigTrasnformations on the "Exists'. Occasionally I've run afoul of the accessing an ItemGroup Item's metadata at the wrong time, I don't know if I've tried to run an "exists" check on something built build up from ItemGroup metadata but looks like it would be desirable.

Exclude/Remove Performance

You are absolutely right that removing a potentially large number of items from a large ItemGroup using the <SomeLargeItemGroupLikeNone Remove=(@SomeSpecificItemGroup)" /> will become non-performant. So for ItemGroups that are potentially large (like None or Compile) it is definitely best to use Excludes when available, or to Remove by IdentityGlob ("**.*" or some such).

Looking back on it, I think I now see what I didn't initially, is that the new Core Web and Core Razor Sdk projects:

• First remove almost everything from the None group by using <None Remove="wwwroot/**" /> which efficiently ensures that None is a small group (Microsoft.NET.Sdk.Razor\Sdk\Sdk.Razor.StaticAssets.ProjectSystem.props)
• Only After that then use the <None Remove="@(_WebToolingArtifacts)" /> because the ItemGroups in play will be very small. (Microsoft.NET.Sdk.Web.ProjectSystem/targets/Microsoft.NET.Sdk.Web.DefaultItems.props)

I've been trying to brainstorm along the lines of what is the best "balancing of separation from expectations"... aka how best and most easily explained/followed "do something different than what is expected" if this Sdk were going to try to support a EnableDefaultWebStaticAsset flavor of defaulting items into Content.

• Have a EnableDefaultNoneItems property that is not taken into account when collecting to an ItemGroup for certain reasons (aka adding the build configuration based web..config files, and launchsettings.json file).
  • I didn't see any examples of any Sdk's doing this
  • Core Web project Sdk even prevents the default inclusion of launchsettings.json and other project related files when <EnableDefaultNoneItems>False</EnableDefaultNoneItems>
  • So doing <EnableDefaultNoneItems>False</EnableDefaultNoneItems> and then including LaunchSettings.json (or even the Web..config files) anyway seems not the best
• Have a DefaultItemExcludes property that is not used when collecting certain ItemGroups
  • I didn't see any other examples outside of this one, that takes this approach... pretty much every "file" inclusion runs with Exclude=$(DefaultItemExcludes);$(DefaultExcludesInProjectFolder)
  • It is kind of this SDk's current strategy... build up a DefaultWebFormsItemExcludes property to attempt to mimic the OOB DefaultItemExcludes property (Intermediateouteputpath, outputpath. etc). And then use this property and not the traditional one when doing the globbing for items, which allows items that would have been excluded by the standard DefaultItemExcludes property to be included with the custom Item inclusion

I think the way it's being done is probably the best, or pretty close. But, it sure would have been so nice if Microsoft had established ItemGroup specific excludes that would have allowed us to exclude things from some ItemGroups and not others.

DynamicFileNesting Performance

Those issues you shared about DynamicFileNesting capability are interesting reads, and I had never come across them before. My previous incarnations had attempted to make an Sdk package that mimicked closely the structure of the Core Web and Core Razor Sdk's that still pulled in snippets of the old:

• Microsoft.WebApplication.targets for packagin/publishing
• Copy/Paste of the "hardcoded" property values that came with every OOB Web Application project created with VS

And since it was a capability pulled in by the web and Razor Sdk's I just pulled it in too.

I'd support undoing that ProjectCapability if it is known to be troublesome. especially since not having them appear nested is really just an appearance thing (even if it is a nice thing), and doesn't alter the build/package time of anything.

I should probably create a dummy project that has lots of files that need this DynamicFileNesting to see what the performance impact of turning it on and off. And that might be a good practice going forward, to "load test" the various aspects of the Sdk.

Clearing out ItemGroups

I guess old habits die hard, and agree that this is not needed in new Sdk style projects. I am just used to writing MsBuild customizations that would function correctly in both Legacy and Sdk style projects. In Non-Sdk project styles, if a "file" is present in more than one item group by a certain point in the build, it would generate an error. And so my habit was to "collect" items I desired into whatever transient ItemGroup, use the transient group to do what I wanted (default metadata, calculated metadata, generate relative paths, etc), transfer them to the Permanent ItemGroup (Content/None/Compile/Etc) if desired, and lastly clear out the transient group as soon as I was done with it.

[leusbj]

On a tangentially related note, you made comment about potentially adding more things to the documentation.

I did have a couple of questions about your preferences

• Are you actively desiring that the WIki and the GitHubPages (maintained via the Docs directory of .md files) be synced and kept up to date, or is the focus mostly on the GithubPages version of the documentation?
• As changes are made to the various projects, that might potentially alter a project template, Would you like changes to the templates to occur at the same time, or independently... and likewise any documentation of the Templates projects as well?
• Do you have preferences global.json version number, should it be kept up to date pointing at the "most recent" version available for common consumers, or prefer that it remain consistent for period of time and updated occasionally?

[CZEMacLeod]

Lol - These are all really good questions which I am not sure I have the answers to!

• I think I had the wiki enabled first then found the docs folder / github pages route and thought it might be better to be able to keep the docs in sync with the changes as they were made. So, I think the GithubPages is the focus - but I don't really do a lot of open-source stuff like this and would appreciate any feedback if you have more experience.
• Generally, changes to the SDK should include any changes to the samples, and to the templates 'in one'. Any new properties etc. should also be added to the documentation where possible.
• I think the global.json should be kept fairly up to date - see above about updating the samples and templates. I should really update the build SDK to use 6.0.x too as 5.0.x is now obsolete.

There are a couple of things I would like to do around the build/release process, but I figured it was good enough, and I don't do that many releases.

1. Making the package included readmes sync with the docs version
2. Automatically patching the templates project files to have the same version number as is currently being built. At the moment I tend to do 2 releases which are essentially the same for the main SDK, where the second has the version numbers patched to +1 from the last build so it all syncs up.
3. Same as above but to patch the global.json
4. Commit the patches etc from above (skipping the CI build of course)

[CZEMacLeod]

Is it worth moving this discussion to a new thread?