Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement zoneTracker for coordinating downscales across statefulsets using a file in object storage for the prepare_downscale admission webhook #96

Closed
wants to merge 15 commits into from
Closed

Implement zoneTracker for coordinating downscales across statefulsets using a file in object storage for the prepare_downscale admission webhook #96

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
f5be68c
Implement zoneTracker for coordinating downscales across statefulsets…
JordanRushing Nov 14, 2023
7d1958f
Merge branch 'main' into prep-downscale-last-downscale-objstore
JordanRushing Nov 14, 2023
60b37ec
Expand tests; fix deadlocks in zoneTracker; add logic to create an in…
JordanRushing Nov 14, 2023
bd9ef2d
Add zoneTracker support for s3; validate new config flags for object …
JordanRushing Nov 28, 2023
b11ec77
Fix s3 config env variable names
JordanRushing Nov 28, 2023
33dd5e5
Add bucket client support for gcs, azure; update config flag validati…
JordanRushing Dec 1, 2023
f2b53ef
Add endpoint to Azure bucket client config; update config validation
JordanRushing Dec 5, 2023
93efd5d
Add additional zoneTracker tests
JordanRushing Dec 6, 2023
7bd2c6d
Remove redundant `defer r.close()` from `loadZones`
JordanRushing Dec 6, 2023
6e4bf58
Remove unused ctx from function definitions; update comment for saveZ…
JordanRushing Dec 13, 2023
0f40551
Merge branch 'main' into prep-downscale-last-downscale-objstore
JordanRushing Dec 14, 2023
0b07920
Resolve merge conflicts with `main`; run `go mod tidy` & `go mod vendor`
JordanRushing Dec 14, 2023
c269119
Convert zoneTracker zones map to use zoneInfo struct instead of strin…
JordanRushing Dec 14, 2023
f62c470
Clean up loadZones function
JordanRushing Dec 14, 2023
72c1cda
Fix setDownscaled to actually update the zoneInfo in the map
JordanRushing Dec 14, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
48 changes: 47 additions & 1 deletion cmd/rollout-operator/main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ import (
"net/http"
"os"
"os/signal"
"strings"
"syscall"
"time"

Expand All @@ -19,6 +20,7 @@ import (
"github.com/prometheus/client_golang/prometheus/promhttp"
"github.com/prometheus/common/model"
"go.uber.org/atomic"
v1 "k8s.io/api/admission/v1"
"k8s.io/client-go/kubernetes"
"k8s.io/client-go/rest"
"k8s.io/client-go/tools/clientcmd"
Expand Down Expand Up @@ -54,6 +56,13 @@ type config struct {
serverSelfSignedCertExpiration model.Duration

updateWebhooksWithSelfSignedCABundle bool

useZoneTracker bool
objectStorageProvider string
bucketName string
objectStorageEndpoint string
objectStorageRegion string
accountName string
}

func (cfg *config) register(fs *flag.FlagSet) {
Expand All @@ -77,6 +86,13 @@ func (cfg *config) register(fs *flag.FlagSet) {
cfg.serverSelfSignedCertExpiration = defaultServerSelfSignedCertExpiration

fs.BoolVar(&cfg.updateWebhooksWithSelfSignedCABundle, "webhooks.update-ca-bundle", true, "Update the CA bundle in the properly labeled webhook configurations with the self-signed certificate (-server-tls.self-signed-cert.enabled should be enabled).")

fs.BoolVar(&cfg.useZoneTracker, "use-zone-tracker", false, "Use the zone tracker to prevent simultaneous downscales in different zones")
fs.StringVar(&cfg.objectStorageProvider, "provider", "", "The object storage provider to use for the zone tracker. Supported values: s3, gcs, azure")
fs.StringVar(&cfg.bucketName, "bucket-name", "", "The name of the bucket to use for the zone tracker")
fs.StringVar(&cfg.objectStorageEndpoint, "endpoint", "", "The endpoint to use for the object storage provider")
fs.StringVar(&cfg.objectStorageRegion, "region", "", "The region to use for the object storage provider")
fs.StringVar(&cfg.accountName, "account-name", "", "The account name to use for the Azure object storage provider")
}

func (cfg config) validate() error {
Expand All @@ -87,6 +103,32 @@ func (cfg config) validate() error {
if (cfg.kubeAPIURL == "") != (cfg.kubeConfigFile == "") {
return errors.New("either configure both Kubernetes API URL and config file or none of them")
}
if cfg.useZoneTracker {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code in this branch belongs better in zone_tracker.go

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know if I agree here, this validates the required config fields to launch with the zt following the pattern we use to validate the required kube config as well.

if cfg.objectStorageProvider == "" {
return errors.New("the object storage provider for the zone tracker has not been specified")
}
if cfg.bucketName == "" {
return errors.New("the bucket name for the zone tracker has not been specified")
}
supportedProviders := []string{"s3", "gcs", "azure"}
supported := false
for _, provider := range supportedProviders {
if cfg.objectStorageProvider == provider {
supported = true
break
}
}
if !supported {
return fmt.Errorf("the provided object storage provider is not supported (currently supported providers: %s)", strings.Join(supportedProviders, ", "))
}
if cfg.objectStorageProvider == "azure" && (cfg.accountName == "" || cfg.objectStorageEndpoint == "") {
return errors.New("the account name for the Azure object storage provider has not been specified")
}
if cfg.objectStorageProvider == "s3" && (cfg.objectStorageEndpoint == "" || cfg.objectStorageRegion == "") {
return errors.New("the endpoint and region for the S3 object storage provider have not been specified")
}
jhalterman marked this conversation as resolved.
Show resolved Hide resolved
}

return nil
}

Expand Down Expand Up @@ -186,10 +228,14 @@ func maybeStartTLSServer(cfg config, logger log.Logger, kubeClient *kubernetes.C
check(tlscert.PatchCABundleOnMutatingWebhooks(context.Background(), logger, kubeClient, cfg.kubeNamespace, cert.CA))
}

prepDownscaleAdmitFunc := func(ctx context.Context, logger log.Logger, ar v1.AdmissionReview, api *kubernetes.Clientset) *v1.AdmissionResponse {
return admission.PrepareDownscale(ctx, logger, ar, api, cfg.useZoneTracker, cfg.objectStorageProvider, cfg.bucketName, cfg.objectStorageEndpoint, cfg.objectStorageRegion, cfg.accountName)
}

tlsSrv, err := newTLSServer(cfg.serverTLSPort, logger, cert, metrics)
check(errors.Wrap(err, "failed to create tls server"))
tlsSrv.Handle(admission.NoDownscaleWebhookPath, admission.Serve(admission.NoDownscale, logger, kubeClient))
tlsSrv.Handle(admission.PrepareDownscaleWebhookPath, admission.Serve(admission.PrepareDownscale, logger, kubeClient))
tlsSrv.Handle(admission.PrepareDownscaleWebhookPath, admission.Serve(prepDownscaleAdmitFunc, logger, kubeClient))
check(errors.Wrap(tlsSrv.Start(), "failed to start tls server"))
}

Expand Down
72 changes: 58 additions & 14 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,12 +17,57 @@ require (
github.com/sirupsen/logrus v1.9.3
github.com/stretchr/testify v1.8.4
go.uber.org/atomic v1.11.0
golang.org/x/sync v0.4.0
golang.org/x/sync v0.5.0
k8s.io/api v0.28.3
k8s.io/apimachinery v0.28.3
k8s.io/client-go v0.28.3
)

require (
cloud.google.com/go v0.110.10 // indirect
cloud.google.com/go/compute v1.23.3 // indirect
cloud.google.com/go/compute/metadata v0.2.3 // indirect
cloud.google.com/go/iam v1.1.5 // indirect
cloud.google.com/go/storage v1.35.1 // indirect
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.8.0 // indirect
github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.4.0 // indirect
github.com/Azure/azure-sdk-for-go/sdk/internal v1.3.0 // indirect
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v0.5.1 // indirect
github.com/AzureAD/microsoft-authentication-library-for-go v1.1.1 // indirect
github.com/aws/aws-sdk-go-v2 v1.16.0 // indirect
github.com/aws/aws-sdk-go-v2/config v1.15.1 // indirect
github.com/aws/aws-sdk-go-v2/credentials v1.11.0 // indirect
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.12.1 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.7 // indirect
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.4.1 // indirect
github.com/aws/aws-sdk-go-v2/internal/ini v1.3.8 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.9.1 // indirect
github.com/aws/aws-sdk-go-v2/service/sso v1.11.1 // indirect
github.com/aws/aws-sdk-go-v2/service/sts v1.16.1 // indirect
github.com/aws/smithy-go v1.11.1 // indirect
github.com/dustin/go-humanize v1.0.1 // indirect
github.com/efficientgo/core v1.0.0-rc.0.0.20221201130417-ba593f67d2a4 // indirect
github.com/golang-jwt/jwt/v5 v5.0.0 // indirect
github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
github.com/google/s2a-go v0.1.7 // indirect
github.com/googleapis/enterprise-certificate-proxy v0.3.2 // indirect
github.com/googleapis/gax-go/v2 v2.12.0 // indirect
github.com/klauspost/cpuid/v2 v2.2.5 // indirect
github.com/kylelemons/godebug v1.1.0 // indirect
github.com/minio/md5-simd v1.1.2 // indirect
github.com/minio/minio-go/v7 v7.0.61 // indirect
github.com/minio/sha256-simd v1.0.1 // indirect
github.com/pkg/browser v0.0.0-20210911075715-681adbf594b8 // indirect
github.com/rs/xid v1.5.0 // indirect
go.opencensus.io v0.24.0 // indirect
golang.org/x/xerrors v0.0.0-20231012003039-104605ab7028 // indirect
google.golang.org/api v0.150.0 // indirect
google.golang.org/genproto v0.0.0-20231106174013-bbf56f31fb17 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20231106174013-bbf56f31fb17 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20231106174013-bbf56f31fb17 // indirect
google.golang.org/grpc v1.59.0 // indirect
)

require (
github.com/Azure/go-ansiterm v0.0.0-20230124172434-306776ec8161 // indirect
github.com/Microsoft/go-winio v0.6.1 // indirect
Expand Down Expand Up @@ -59,10 +104,10 @@ require (
github.com/golang/protobuf v1.5.3 // indirect
github.com/goodhosts/hostsfile v0.1.1 // indirect
github.com/google/gnostic-models v0.6.8 // indirect
github.com/google/go-cmp v0.5.9 // indirect
github.com/google/go-cmp v0.6.0 // indirect
github.com/google/go-containerregistry v0.16.1 // indirect
github.com/google/gofuzz v1.2.0 // indirect
github.com/google/uuid v1.3.1 // indirect
github.com/google/uuid v1.4.0 // indirect
github.com/hashicorp/errwrap v1.1.0 // indirect
github.com/hashicorp/hcl v1.0.0 // indirect
github.com/imdario/mergo v0.3.14 // indirect
Expand Down Expand Up @@ -103,6 +148,7 @@ require (
github.com/spf13/pflag v1.0.5 // indirect
github.com/spf13/viper v1.16.0 // indirect
github.com/subosito/gotenv v1.4.2 // indirect
github.com/thanos-io/objstore v0.0.0-20231123170144-bffedaa58acb
github.com/theupdateframework/notary v0.7.0 // indirect
github.com/uber/jaeger-client-go v2.28.0+incompatible // indirect
github.com/uber/jaeger-lib v2.2.0+incompatible // indirect
Expand All @@ -112,22 +158,20 @@ require (
github.com/xeipuuv/gojsonschema v1.2.0 // indirect
go.uber.org/multierr v1.8.0 // indirect
go4.org/netipx v0.0.0-20230728184502-ec4c8b891b28 // indirect
golang.org/x/crypto v0.14.0 // indirect
golang.org/x/crypto v0.15.0 // indirect
golang.org/x/mod v0.10.0 // indirect
golang.org/x/net v0.17.0 // indirect
golang.org/x/oauth2 v0.12.0 // indirect
golang.org/x/sys v0.13.0 // indirect
golang.org/x/term v0.13.0 // indirect
golang.org/x/text v0.13.0 // indirect
golang.org/x/time v0.3.0 // indirect
golang.org/x/net v0.18.0 // indirect
golang.org/x/oauth2 v0.14.0 // indirect
golang.org/x/sys v0.14.0 // indirect
golang.org/x/term v0.14.0 // indirect
golang.org/x/text v0.14.0 // indirect
golang.org/x/time v0.4.0 // indirect
golang.org/x/tools v0.9.1 // indirect
google.golang.org/appengine v1.6.7 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20230822172742-b8732ec3820d // indirect
google.golang.org/grpc v1.59.0 // indirect
google.golang.org/appengine v1.6.8 // indirect
google.golang.org/protobuf v1.31.0 // indirect
gopkg.in/inf.v0 v0.9.1 // indirect
gopkg.in/ini.v1 v1.67.0 // indirect
gopkg.in/yaml.v2 v2.4.0 // indirect
gopkg.in/yaml.v2 v2.4.0
gopkg.in/yaml.v3 v3.0.1 // indirect
k8s.io/klog/v2 v2.100.1 // indirect
k8s.io/kube-openapi v0.0.0-20230717233707-2695361300d9 // indirect
Expand Down
Loading