Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

✨ Initialize hub with aws irsa #465

Merged
merged 1 commit into from
Jan 21, 2025
+217 −10
Merged

✨ Initialize hub with aws irsa #465

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,10 +28,10 @@ require (
k8s.io/klog/v2 v2.130.1
k8s.io/kubectl v0.31.1
k8s.io/utils v0.0.0-20240921022957-49e7df575cb6
open-cluster-management.io/api v0.15.1-0.20250109024121-1a5e25a78a43
open-cluster-management.io/api v0.15.1-0.20250116010516-3a595d6a4e40
open-cluster-management.io/cluster-proxy v0.4.0
open-cluster-management.io/managed-serviceaccount v0.6.0
open-cluster-management.io/ocm v0.15.1-0.20250116085531-34275ef1eac8
open-cluster-management.io/ocm v0.15.1-0.20250120013556-eeb4ab31d5ab
open-cluster-management.io/sdk-go v0.15.1-0.20241125015855-1536c3970f8f
sigs.k8s.io/apiserver-network-proxy v0.29.0
sigs.k8s.io/apiserver-network-proxy/konnectivity-client v0.30.3
Expand Down
8 changes: 4 additions & 4 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -557,14 +557,14 @@ k8s.io/kubectl v0.31.1 h1:ih4JQJHxsEggFqDJEHSOdJ69ZxZftgeZvYo7M/cpp24=
k8s.io/kubectl v0.31.1/go.mod h1:aNuQoR43W6MLAtXQ/Bu4GDmoHlbhHKuyD49lmTC8eJM=
k8s.io/utils v0.0.0-20240921022957-49e7df575cb6 h1:MDF6h2H/h4tbzmtIKTuctcwZmY0tY9mD9fNT47QO6HI=
k8s.io/utils v0.0.0-20240921022957-49e7df575cb6/go.mod h1:OLgZIPagt7ERELqWJFomSt595RzquPNLL48iOWgYOg0=
open-cluster-management.io/api v0.15.1-0.20250109024121-1a5e25a78a43 h1:9kgKRQQHMGNM1t+J+OrmF7hgZmND9kRwyRVnHIULzqw=
open-cluster-management.io/api v0.15.1-0.20250109024121-1a5e25a78a43/go.mod h1:9erZEWEn4bEqh0nIX2wA7f/s3KCuFycQdBrPrRzi0QM=
open-cluster-management.io/api v0.15.1-0.20250116010516-3a595d6a4e40 h1:LckTHZ68rcy3hDFu6wa7BVOJ9wbWItJLZXmi0bpMyh8=
open-cluster-management.io/api v0.15.1-0.20250116010516-3a595d6a4e40/go.mod h1:9erZEWEn4bEqh0nIX2wA7f/s3KCuFycQdBrPrRzi0QM=
open-cluster-management.io/cluster-proxy v0.4.0 h1:rm0UDaDWe3/P3xLzwqdHtqNksKwSzsic02MkrEe6BnM=
open-cluster-management.io/cluster-proxy v0.4.0/go.mod h1:gTvfDHAhGezhdg4BD3ECBn6jbg2Y5PbHhV2ceW5nrB0=
open-cluster-management.io/managed-serviceaccount v0.6.0 h1:qIi5T9WQJBuoGqnYGIktXbtqfQoiN2H9XU2P/6lAQiw=
open-cluster-management.io/managed-serviceaccount v0.6.0/go.mod h1:G4LUTbZiyrB8c0+rqi/xnDmGlsg7Rdr4T7MPLCWhyQI=
open-cluster-management.io/ocm v0.15.1-0.20250116085531-34275ef1eac8 h1:IDjk8EeKajwqezVM1eDNYPHyaJx4V0N/sZoSAVhIUJk=
open-cluster-management.io/ocm v0.15.1-0.20250116085531-34275ef1eac8/go.mod h1:daPkqFxkVqKb4O8UTX+7jCyEcJWarGOG7uDie9rFfck=
open-cluster-management.io/ocm v0.15.1-0.20250120013556-eeb4ab31d5ab h1:DY4DSDQUEoVQ6fCda7nSYetJRhvkyoiHPLyMppL/a8w=
open-cluster-management.io/ocm v0.15.1-0.20250120013556-eeb4ab31d5ab/go.mod h1:Mfg6rf0CylcnY5y8zJB99ClbMUMpAAUa22Rv+3ct5Lg=
open-cluster-management.io/sdk-go v0.15.1-0.20241125015855-1536c3970f8f h1:zeC7QrFNarfK2zY6jGtd+mX+yDrQQmnH/J8A7n5Nh38=
open-cluster-management.io/sdk-go v0.15.1-0.20241125015855-1536c3970f8f/go.mod h1:fi5WBsbC5K3txKb8eRLuP0Sim/Oqz/PHX18skAEyjiA=
oras.land/oras-go v1.2.5 h1:XpYuAwAb0DfQsunIyMfeET92emK8km3W4yEzZvUbsTo=
Expand Down
7 changes: 7 additions & 0 deletions pkg/cmd/init/cmd.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,9 @@ import (
var example = `
# Init the hub
%[1]s init

# Initialize the hub cluster with the type of authentication. Either or both of csr,awsirsa
%[1]s init --registration-auth awsirsa --registration-auth csr --hubClusterArn arn:aws:eks:us-west-2:123456789012:cluster/hub-cluster1
`

// NewCmd ...
Expand Down Expand Up @@ -78,6 +81,10 @@ func NewCmd(clusteradmFlags *genericclioptionsclusteradm.ClusteradmFlags, stream
_ = clusterManagerSet.SetAnnotation("singleton-name", "singletonSet", []string{})
o.Helm.AddFlags(singletonSet)
cmd.Flags().AddFlagSet(singletonSet)
cmd.Flags().StringArrayVar(&o.registrationAuth, "registration-auth", []string{},
"The type of authentication to use for registering and authenticating with hub. Only csr and awsirsa are accepted as valid inputs. This flag can be repeated to specify multiple authentication types.")
cmd.Flags().StringVar(&o.hubClusterArn, "hub-cluster-arn", "",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you would need to validate this value

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thr validation for this is already added to cluster-manager spec and the validation error is already propagated back to the clusteradm consumer even now.

Is that sufficient?

"The hubCluster ARN to be passed if awsirsa is one of the registrationAuths and the cluster name in EKS kubeconfig doesn't contain hubClusterArn")

return cmd
}
50 changes: 50 additions & 0 deletions pkg/cmd/init/exec.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ package init
import (
"context"
"fmt"
"k8s.io/apimachinery/pkg/util/sets"
"os"
"time"

Expand Down Expand Up @@ -70,10 +71,15 @@ func (o *Options) complete(cmd *cobra.Command, args []string) (err error) {
},
Tag: bundleVersion.OCM,
}
registrationDrivers, err := getRegistrationDrivers(o)
if err != nil {
return err
}
o.clusterManagerChartConfig.ClusterManager = chart.ClusterManagerConfig{
RegistrationConfiguration: operatorv1.RegistrationHubConfiguration{
FeatureGates: genericclioptionsclusteradm.ConvertToFeatureGateAPI(
genericclioptionsclusteradm.HubMutableFeatureGate, ocmfeature.DefaultHubRegistrationFeatureGates),
RegistrationDrivers: registrationDrivers,
},
WorkConfiguration: operatorv1.WorkConfiguration{
FeatureGates: genericclioptionsclusteradm.ConvertToFeatureGateAPI(
Expand Down Expand Up @@ -144,6 +150,13 @@ func (o *Options) validate() error {
return fmt.Errorf("registry should not be empty")
}

validRegistrationDriver := sets.New[string]("csr", "awsirsa")
for _, driver := range o.registrationAuth {
if !validRegistrationDriver.Has(driver) {
return fmt.Errorf("only csr and awsirsa are valid drivers")
}
}

// If --wait is set, some information during initialize process will print to output, the output would not keep
// machine readable, so this behavior should be disabled
if o.wait && o.output != "text" {
Expand Down Expand Up @@ -353,3 +366,40 @@ func (o *Options) deploySingletonControlplane(kubeClient kubernetes.Interface) e
}
return nil
}

func getRegistrationDrivers(o *Options) ([]operatorv1.RegistrationDriverHub, error) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could we make it simpler? we should validate and return error when a driver name is not allowed in advance.

for _, driver := range o.registrationAuth:
    registrationDriver :=  operatorv1.RegistrationDriverHub{AuthType: driver}
   if driver == "awisra" {
      ...
   }
   registrationDrivers = append(registrationDrivers, registrationDriver)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

registrationDrivers := []operatorv1.RegistrationDriverHub{}
var registrationDriver operatorv1.RegistrationDriverHub

for _, driver := range o.registrationAuth {
if driver == "csr" {
registrationDriver = operatorv1.RegistrationDriverHub{AuthType: driver}
} else if driver == "awsirsa" {
hubClusterArn, err := getHubClusterArn(o)
if err != nil {
return registrationDrivers, err
}
registrationDriver = operatorv1.RegistrationDriverHub{AuthType: driver, HubClusterArn: hubClusterArn}
}
registrationDrivers = append(registrationDrivers, registrationDriver)
}

return registrationDrivers, nil
}

func getHubClusterArn(o *Options) (string, error) {
hubClusterArn := o.hubClusterArn
if hubClusterArn == "" {
rawConfig, err := o.ClusteradmFlags.KubectlFactory.ToRawKubeConfigLoader().RawConfig()
if err != nil {
klog.Errorf("unable to load hub cluster kubeconfig: %v", err)
return "", err
}
hubClusterArn = rawConfig.Contexts[rawConfig.CurrentContext].Cluster
if hubClusterArn == "" {
klog.Errorf("hubClusterArn has empty value in kubeconfig")
return "", fmt.Errorf("unable to retrieve hubClusterArn from kubeconfig")
}
}
return hubClusterArn, nil
}
6 changes: 6 additions & 0 deletions pkg/cmd/init/options.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,12 @@ type Options struct {
output string

Streams genericiooptions.IOStreams

// The type of authentication to use for initializing the hub cluster
registrationAuth []string
// The optional ARN to pass if awsirsa is one of the registrationAuths
// and the cluster name in EKS kubeconfig doesn't contain hubClusterArn
hubClusterArn string
}

func newOptions(clusteradmFlags *genericclioptionsclusteradm.ClusteradmFlags, streams genericiooptions.IOStreams) *Options {
Expand Down
35 changes: 35 additions & 0 deletions test/e2e/clusteradm/init_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,41 @@ var _ = ginkgo.Describe("test clusteradm with bootstrap token in singleton mode"
gomega.Expect(err).NotTo(gomega.HaveOccurred())
gomega.Expect(len(cm.Spec.RegistrationConfiguration.FeatureGates)).Should(gomega.Equal(1))

// TODO: E2e test is not recognizing the newly added flags. Uncomment below test once the problem is fixed.
//err = e2e.Clusteradm().Init(
// "--use-bootstrap-token",
// "--context", e2e.Cluster().Hub().Context(),
// "--bundle-version=latest",
// "--registration-auth awsirsa",
// "--hub-cluster-arn arn:aws:eks:us-west-2:123456789012:cluster/hub-cluster1",
//)
//gomega.Expect(err).NotTo(gomega.HaveOccurred(), "clusteradm init error")
//
//cm, err = operatorClient.OperatorV1().ClusterManagers().Get(context.TODO(), "cluster-manager", metav1.GetOptions{})
//gomega.Expect(err).NotTo(gomega.HaveOccurred())
//// Ensure that when only awsirsa is passed as registration-auth only awsirsa driver is available
//gomega.Expect(len(cm.Spec.RegistrationConfiguration.RegistrationDrivers)).Should(gomega.Equal(1))
//gomega.Expect(cm.Spec.RegistrationConfiguration.RegistrationDrivers[0].AuthType).Should(gomega.Equal("awsirsa"))
//
//err = e2e.Clusteradm().Init(
// "--use-bootstrap-token",
// "--context", e2e.Cluster().Hub().Context(),
// "--bundle-version=latest",
// "--registration-auth awsirsa",
// "--registration-auth csr",
// "--hub-cluster-arn arn:aws:eks:us-west-2:123456789012:cluster/hub-cluster1",
//)
//gomega.Expect(err).NotTo(gomega.HaveOccurred(), "clusteradm init error")
//
//cm, err = operatorClient.OperatorV1().ClusterManagers().Get(context.TODO(), "cluster-manager", metav1.GetOptions{})
//gomega.Expect(err).NotTo(gomega.HaveOccurred())
//// Ensure that awsirsa and csr is passed as registration-auth both the values are set.
//gomega.Expect(len(cm.Spec.RegistrationConfiguration.RegistrationDrivers)).Should(gomega.Equal(2))
//gomega.Expect(cm.Spec.RegistrationConfiguration.RegistrationDrivers[0].AuthType).Should(gomega.Equal("csr"))
//gomega.Expect(cm.Spec.RegistrationConfiguration.RegistrationDrivers[1].AuthType).Should(gomega.Equal("awsirsa"))
//gomega.Expect(cm.Spec.RegistrationConfiguration.RegistrationDrivers[1].HubClusterArn).
// Should(gomega.Equal("arn:aws:eks:us-west-2:123456789012:cluster/hub-cluster1"))

err = e2e.Clusteradm().Init(
"--use-bootstrap-token",
"--context", e2e.Cluster().Hub().Context(),
Expand Down
4 changes: 2 additions & 2 deletions vendor/modules.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1239,7 +1239,7 @@ k8s.io/utils/pointer
k8s.io/utils/ptr
k8s.io/utils/strings/slices
k8s.io/utils/trace
# open-cluster-management.io/api v0.15.1-0.20250109024121-1a5e25a78a43
# open-cluster-management.io/api v0.15.1-0.20250116010516-3a595d6a4e40
## explicit; go 1.22.0
open-cluster-management.io/api/addon/v1alpha1
open-cluster-management.io/api/client/addon/clientset/versioned
Expand Down Expand Up @@ -1282,7 +1282,7 @@ open-cluster-management.io/managed-serviceaccount/pkg/generated/clientset/versio
open-cluster-management.io/managed-serviceaccount/pkg/generated/clientset/versioned/scheme
open-cluster-management.io/managed-serviceaccount/pkg/generated/clientset/versioned/typed/authentication/v1alpha1
open-cluster-management.io/managed-serviceaccount/pkg/generated/clientset/versioned/typed/authentication/v1beta1
# open-cluster-management.io/ocm v0.15.1-0.20250116085531-34275ef1eac8
# open-cluster-management.io/ocm v0.15.1-0.20250120013556-eeb4ab31d5ab
## explicit; go 1.22.5
open-cluster-management.io/ocm/deploy/cluster-manager/chart
open-cluster-management.io/ocm/deploy/klusterlet/chart
Expand Down
25 changes: 25 additions & 0 deletions ...t.io/api/operator/v1/0000_01_operator.open-cluster-management.io_clustermanagers.crd.yaml
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

22 changes: 22 additions & 0 deletions vendor/open-cluster-management.io/api/operator/v1/types_clustermanager.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

21 changes: 21 additions & 0 deletions vendor/open-cluster-management.io/api/operator/v1/zz_generated.deepcopy.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

14 changes: 12 additions & 2 deletions vendor/open-cluster-management.io/api/operator/v1/zz_generated.swagger_doc_generated.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

25 changes: 25 additions & 0 deletions ...cluster-manager/crds/0000_01_operator.open-cluster-management.io_clustermanagers.crd.yaml
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 4 additions & 0 deletions ...gement.io/ocm/deploy/cluster-manager/chart/cluster-manager/templates/cluster_manager.yaml
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions ...r/open-cluster-management.io/ocm/deploy/cluster-manager/chart/cluster-manager/values.yaml
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading