Update documentation and allow asoctl to handle deprecated trustedacc…

…essrolebinding (#4403)

* Some small documentation updates

* Small asoctl update to support deprecating other API versions

We should probably refactor this more, but this solves the immediate
problem.

docs/hugo/content/contributing/create-a-new-release.md

@@ -4,20 +4,21 @@ title: Creating a new release
 
 1. Ensure that the [controller CVE scan action](https://github.com/Azure/azure-service-operator/actions/workflows/scan-controller-image.yaml) is green.
 2. Ensure that the [pre-release upgrade tests](https://github.com/Azure/azure-service-operator/actions/workflows/pre-release-tests.yaml) are passing.
-3. Go to the [releases](https://github.com/Azure/azure-service-operator/releases) page and draft a new release.
-4. In the tag dropdown, type the name of the new tag you'd like to create (it should match the pattern of previous releases tags, for example: `v2.0.0-beta.3`). The release target should be `main` (the default).
-5. Use the GitHub "auto-generate release notes" button to generate a set of release notes to work with. You will need to clean this up quite a bit before actually publishing it.
-6. Write a "Release Notes" section. You can edit the autogenerated section as a start. You can also look through the commits between the last release and now: 
+3. Ensure that [experimental release](https://github.com/Azure/azure-service-operator/releases/tag/experimental) passed on the most recent commit.
+4. Go to the [releases](https://github.com/Azure/azure-service-operator/releases) page and draft a new release.
+5. In the tag dropdown, type the name of the new tag you'd like to create (it should match the pattern of previous releases tags, for example: `v2.10.0`). The release target should be `main` (the default).
+6. Use the GitHub "auto-generate release notes" button to generate a set of release notes to work with. You will need to clean this up quite a bit before actually publishing it.
+7. Write a "Release Notes" section. You can edit the autogenerated section as a start. You can also look through the commits between the last release and now: 
 
    ``` bash
    git log v2.4.0..main
    ```
 
-7. Publish the release. This will automatically trigger a GitHub action to build and publish an updated Docker image with the latest manager changes.
-8. Ensure that the action associated with your release finishes successfully.
-9. [Update our documentation](#update-resource-documentation) to move resources listed under "Next Release" to the heading "Released".
-10. [Update the ROADMAP](#update-roadmap) to reflect the new release.
-11. [Update Breaking Changes](#update-breaking-changes) if there are any.
+8. Publish the release. This will automatically trigger a GitHub action to build and publish an updated Docker image with the latest manager changes.
+9. Ensure that the action associated with your release finishes successfully.
+10. [Update our documentation](#update-resource-documentation) to move resources listed under "Next Release" to the heading "Released".
+11. [Update the ROADMAP](#update-roadmap) to reflect the new release.
+12. [Update Breaking Changes](#update-breaking-changes) if there are any.
 
 ## Catalog breaking changes
 

docs/hugo/content/guide/breaking-changes/breaking-changes-v2.9.0.md

@@ -6,16 +6,26 @@ weight: -25  # This should be 5 lower than the previous breaking change document
 
 # DelegatedManagedIdentityResourceId is now a secret
 
-The RoleAssignment property `.spec.delegatedManagedIdentityResourceId` has been changed from a string to a SecretReference and renamed to `.spec.delegatedManagedIdentityResourceReference`.
+The RoleAssignment property `.spec.delegatedManagedIdentityResourceId` has been changed from a string to a 
+SecretReference and renamed to `.spec.delegatedManagedIdentityResourceReference`.
 
 We try to avoid breaking changes, but in this case, allowing raw passwords in the spec is a security 
 problem and as such we've decided to make a break to correct this issue.
 
-**Action required:** If the `authorization.azure.com/RoleAssignment` resource is used in your cluster, and if you are using the `DelegatedManagedIdentityResourceId` property, you'll need to change your resource to use the new `DelegatedManagedIdentityResourceReference` property, pulling the value from a secret.
+**Action required:** If the `authorization.azure.com/RoleAssignment` resource is used in your cluster, and if you are 
+using the `DelegatedManagedIdentityResourceId` property, you'll need to change your resource to use the new 
+`DelegatedManagedIdentityResourceReference` property, pulling the value from a secret.
 
 # ContainerService version v1api20230202preview has been deleted
 
-The AKS preview version `2023-02-02-preview` has been deprecated by AKS and is no longer supported. We've removed it from the operator to prevent unrecoverable errors from occurring.
+The AKS preview version `2023-02-02-preview` has been deprecated by AKS and is no longer supported. We've removed it 
+from the operator to prevent unrecoverable errors from occurring.
 
-**Action required:** If you are using this specific version, change your resources to use a different version _before_ you upgrade to v2.9.0. 
+**Action required:** 
 
+- If you are using this specific version, change your resources to use a different version _before_ you upgrade to v2.9.0.
+- Check if the `trustedaccessrolebindings.containerservice.azure.com` CRD is installed. If it is, check if the
+  `status.storedVersions` field contains the `v1api20230202previewstorage` API. 
+  - If it doesn't, no action is needed.
+  - If it **does**, download the latest [experimental](https://github.com/Azure/azure-service-operator/releases/tag/experimental)
+    or v2.11.0+ asoctl and run `asoctl clean crds`.

docs/hugo/content/guide/frequently-asked-questions.md

@@ -172,6 +172,33 @@ env:
         name: identity-details
 ```
 
+You can allow the other environment variables, volumes, and volume mounts to be injected automatically by the 
+[Azure Workload Identity webhook](https://azure.github.io/azure-workload-identity/docs/installation/mutating-admission-webhook.html),
+or you can avoid running the Azure Workload Identity webhook entirely, but doing so requires that you manually 
+include the `azure-identity` volume and volumeMount, as well as set the `AZURE_TENANT_ID` variable 
+alongside `AZURE_CLIENT_ID` in every pod that needs workload identity.
+
+Sample VolumeMount:
+```yaml
+volumeMounts:
+  - mountPath: /var/run/secrets/azure/tokens/azure-identity-token
+    name: azure-identity-token
+    readOnly: true
+```
+
+Sample Volume:
+```yaml
+volumes:
+- name: azure-identity-token
+  projected:
+    defaultMode: 420
+    sources:
+      - serviceAccountToken:
+          audience: api://AzureADTokenExchange
+          expirationSeconds: 3600
+          path: azure-identity
+```
+
 ### How can I feed the output of one resource into a parameter for the next?
 
 The answer changes a bit depending on the nature of the parameter.

v2/cmd/asoctl/internal/crd/cleaner.go

@@ -93,17 +93,15 @@ func (c *Cleaner) Run(ctx context.Context) error {
 
 	var updated int
 	var asoCRDsSeen int
-	deprecatedVersionRegexp := regexp.MustCompile(`((v1alpha1api|v1beta)\d{8}(preview)?(storage)?|v1beta1)`) // handcrafted (non-ARM) resources have v1beta1 version
 
 	for _, crd := range crds {
-		crd := crd
-
 		asoCRDsSeen++
-		newStoredVersions, deprecatedVersion := removeMatchingStoredVersions(crd.Status.StoredVersions, deprecatedVersionRegexp)
+		newStoredVersions, deprecatedVersions := getDeprecatedStorageVersions(crd)
 
 		// If there is no new version found other than the matched version, we short circuit here, as there is no updated version found in the CRDs
 		if len(newStoredVersions) <= 0 {
-			return errors.New(fmt.Sprintf("it doesn't look like your version of ASO is one that supports deprecating version %q. Have you upgraded ASO yet?", deprecatedVersion))
+			// TODO: test?
+			return errors.Errorf("it doesn't look like your version of ASO is one that supports deprecating CRD %q, versions %q. Have you upgraded ASO yet?", crd.Name, deprecatedVersions)
 		}
 
 		// If the slice was not updated, there is no version to deprecate.
@@ -266,18 +264,41 @@ func (c *Cleaner) getObjectsForMigration(ctx context.Context, crd apiextensions.
 	return list, nil
 }
 
-// removeMatchingStoredVersions returns a new list of storedVersions by removing the non-storage matched version
-func removeMatchingStoredVersions(oldVersions []string, versionRegexp *regexp.Regexp) ([]string, string) {
-	newStoredVersions := make([]string, 0, len(oldVersions))
-	var matchedStoredVersion string
-	for _, version := range oldVersions {
-		if versionRegexp.MatchString(version) {
-			matchedStoredVersion = version
+var deprecatedVersionRegexp = regexp.MustCompile(`((v1alpha1api|v1beta)\d{8}(preview)?(storage)?|v1beta1)`) // handcrafted (non-ARM) resources have v1beta1 version
+
+// deprecatedVersionsMap is a map of crd name to a collection of deprecated versions
+var deprecatedVersionsMap = map[string][]string{
+	"trustedaccessrolebindings.containerservice.azure.com": {"v1api20230202previewstorage"},
+}
+
+func isVersionDeprecated(crd apiextensions.CustomResourceDefinition, version string) bool {
+	if deprecatedVersionRegexp.MatchString(version) {
+		return true
+	}
+
+	for _, deprecatedVersion := range deprecatedVersionsMap[crd.Name] {
+		if deprecatedVersion == version {
+			return true
+		}
+	}
+
+	return false
+}
+
+// getDeprecatedStorageVersions returns a new list of storedVersions by removing the deprecated versions
+func getDeprecatedStorageVersions(crd apiextensions.CustomResourceDefinition) ([]string, []string) {
+	storedVersions := crd.Status.StoredVersions
+
+	newStoredVersions := make([]string, 0, len(storedVersions))
+	var removedVersions []string
+	for _, version := range storedVersions {
+		if isVersionDeprecated(crd, version) {
+			removedVersions = append(removedVersions, version)
 			continue
 		}
 
 		newStoredVersions = append(newStoredVersions, version)
 	}
 
-	return newStoredVersions, matchedStoredVersion
+	return newStoredVersions, removedVersions
 }

v2/cmd/asoctl/internal/crd/cleaner_test.go

@@ -7,6 +7,7 @@ package crd
 
 import (
 	"context"
+	"fmt"
 	"testing"
 
 	"github.com/go-logr/logr"
@@ -103,6 +104,38 @@ func Test_CleanDeprecatedCRDVersions_CleansHandcraftedBetaVersion_IfExists(t *te
 	g.Expect(crd.Status.StoredVersions).To(ContainElement(gaVersion))
 }
 
+func Test_CleanDeprecatedCRDVersions_CleansTrustedAccessRoleBindings(t *testing.T) {
+	t.Parallel()
+	g := NewGomegaWithT(t)
+
+	c := makeClientSets()
+
+	oldVersion := "v1api20230202previewstorage"
+	newVersion := "v1api20231001storage"
+
+	// create CRD
+	definition := newCRDWithStoredVersionsAndName(
+		"containerservice.azure.com",
+		"trustedaccessrolebindings",
+		"TrustedAccessRoleBindingList",
+		oldVersion,
+		newVersion)
+
+	_, err := c.fakeApiExtClient.CustomResourceDefinitions().Create(context.TODO(), definition, metav1.CreateOptions{})
+	g.Expect(err).To(BeNil())
+
+	err = c.cleaner.Run(context.TODO())
+	g.Expect(err).To(BeNil())
+
+	crd, err := c.fakeApiExtClient.CustomResourceDefinitions().Get(context.TODO(), definition.Name, metav1.GetOptions{})
+	g.Expect(err).To(BeNil())
+
+	g.Expect(crd.Status.StoredVersions).ToNot(BeNil())
+	g.Expect(crd.Status.StoredVersions).ToNot(BeEquivalentTo(definition.Status.StoredVersions))
+	g.Expect(crd.Status.StoredVersions).ToNot(ContainElement(oldVersion))
+	g.Expect(crd.Status.StoredVersions).To(ContainElement(newVersion))
+}
+
 func Test_MigrateDeprecatedCRDResources_DoesNotMigrateBetaVersion_IfStorage(t *testing.T) {
 	t.Parallel()
 	g := NewGomegaWithT(t)
@@ -335,17 +368,21 @@ func newResourceGroup(name, namespace string) *resources.ResourceGroup {
 }
 
 func newCRDWithStoredVersions(versions ...string) *v1.CustomResourceDefinition {
+	return newCRDWithStoredVersionsAndName("resources.azure.com", "resourcegroups", "ResourceGroup", versions...)
+}
+
+func newCRDWithStoredVersionsAndName(group string, name string, listKind string, versions ...string) *v1.CustomResourceDefinition {
 	definition := &v1.CustomResourceDefinition{
 		ObjectMeta: metav1.ObjectMeta{
-			Name: "resourcegroups.resources.azure.com",
+			Name: fmt.Sprintf("%s.%s", name, group),
 			Labels: map[string]string{
 				"app.kubernetes.io/name": "azure-service-operator",
 			},
 		},
 		Spec: v1.CustomResourceDefinitionSpec{
-			Group: "resources.azure.com",
+			Group: group,
 			Names: v1.CustomResourceDefinitionNames{
-				ListKind: "ResourceGroup",
+				ListKind: listKind,
 			},
 		},
 		Status: v1.CustomResourceDefinitionStatus{