Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kie-issues#2789: [sonataflow-operator] Implement DB Migrator Changes into SonataFlow Operator #2790

Open
wants to merge 13 commits into
base: main
Choose a base branch
from
Open

kie-issues#2789: [sonataflow-operator] Implement DB Migrator Changes into SonataFlow Operator #2790

Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
ab197e9
Begin db migrator changes to sonataflow-operator
rhkp Nov 28, 2024
450006b
address review comments
rhkp Dec 20, 2024
1d57981
address review comments
rhkp Dec 20, 2024
00e2a1d
Run make generate-all command
rhkp Dec 23, 2024
0a0bf99
Address review comments
rhkp Dec 23, 2024
32a4588
Address review comments
rhkp Jan 16, 2025
6042723
Address review comments
rhkp Jan 16, 2025
0f71572
Run make generate-all
rhkp Jan 16, 2025
8bc3c37
Address review comments
rhkp Jan 17, 2025
d3a8809
Run make generate-all
rhkp Jan 17, 2025
c71af46
Add platform level persistence support for db migration
rhkp Jan 21, 2025
25475df
Run make generate-all
rhkp Jan 21, 2025
f0687a7
Run make generate-all
rhkp Jan 21, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 14 additions & 3 deletions packages/sonataflow-operator/api/v1alpha08/sonataflow_persistence_types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,14 @@

package v1alpha08

type DBMigrationStrategyType string

const (
DBMigrationStrategyService DBMigrationStrategyType = "service"
DBMigrationStrategyJob DBMigrationStrategyType = "job"
rhkp marked this conversation as resolved.
Show resolved Hide resolved
DBMigrationStrategyNone DBMigrationStrategyType = "none"
)

// PlatformPersistenceOptionsSpec configures the DataBase in the platform spec. This specification can
// be used by workflows and platform services when they don't provide one of their own.
// +optional
Expand Down Expand Up @@ -54,10 +62,13 @@ type PersistenceOptionsSpec struct {
// +optional
PostgreSQL *PersistencePostgreSQL `json:"postgresql,omitempty"`

// Whether to migrate database on service startup?
// DB Migration approach for service?
rhkp marked this conversation as resolved.
Show resolved Hide resolved
// job: use job based approach
// service: service itself shall migrate the db
// none: no db migration needed
// +optional
// +default: false
MigrateDBOnStartUp bool `json:"migrateDBOnStartUp"`
// +kubebuilder:default:=service
DBMigrationStrategy string `json:"dbMigrationStrategy,omitempty"`
}

// PersistencePostgreSQL configure postgresql connection for service(s).
Expand Down
27 changes: 27 additions & 0 deletions packages/sonataflow-operator/api/v1alpha08/sonataflowplatform_types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,31 @@ const (
PlatformDuplicatedReason = "Duplicated"
)

type DBMigrationStatus string

const (
DBMigrationStatusStarted DBMigrationStatus = "Started"
DBMigrationStatusInProgress DBMigrationStatus = "In-Progress"
DBMigrationStatusSucceeded DBMigrationStatus = "Succeeded"
DBMigrationStatusFailed DBMigrationStatus = "Failed"

MessageDBMigrationStatusStarted string = "Started the database migrations for the services"
MessageDBMigrationStatusInProgress string = "The database migrations for the services are in-progress"
MessageDBMigrationStatusSucceeded string = "The database migrations for the services are successful"
MessageDBMigrationStatusFailed string = "The database migrations for the services have failed"

ReasonDBMigrationStatusStarted string = "Started by SonataFlow operator"
ReasonDBMigrationStatusInProgress string = "The database migration job is in-progress"
ReasonDBMigrationStatusSucceeded string = "The database migration job completed as expected"
ReasonDBMigrationStatusFailed string = "The database may be unreachable, invalid credentials supplied or flyway migration failed. Please check logs for further details."
)

type SonataFlowPlatformDBMigrationPhase struct {
Status DBMigrationStatus `json:"dbMigrationStatus,omitempty"`
Message string `json:"message,omitempty"`
Reason string `json:"reason,omitempty"`
}

// SonataFlowPlatformStatus defines the observed state of SonataFlowPlatform
// +k8s:openapi-gen=true
type SonataFlowPlatformStatus struct {
Expand All @@ -123,6 +148,8 @@ type SonataFlowPlatformStatus struct {
// Triggers list of triggers created for the SonataFlowPlatform
//+operator-sdk:csv:customresourcedefinitions:type=status,displayName="triggers"
Triggers []SonataFlowPlatformTriggerRef `json:"triggers,omitempty"`
//+operator-sdk:csv:customresourcedefinitions:type=status,displayName="dbMigrationStatus"
SonataFlowPlatformDBMigrationPhase *SonataFlowPlatformDBMigrationPhase `json:"sonataFlowPlatformDBMigrationPhase,omitempty"`
}

// SonataFlowPlatformTriggerRef defines a trigger created for the SonataFlowPlatform.
Expand Down
20 changes: 20 additions & 0 deletions packages/sonataflow-operator/api/v1alpha08/zz_generated.deepcopy.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions packages/sonataflow-operator/config/manager/controllers_cfg.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,8 @@ jobsServiceEphemeralImageTag: "docker.io/apache/incubator-kie-kogito-jobs-servic
# The Data Index image to use, if empty the operator will use the default Apache Community one based on the current operator's version
dataIndexPostgreSQLImageTag: "docker.io/apache/incubator-kie-kogito-data-index-postgresql:main"
dataIndexEphemeralImageTag: "docker.io/apache/incubator-kie-kogito-data-index-ephemeral:main"
# The Kogito PostgreSQL DB Migrator image to use (TBD: to replace with apache image)
dbMigratorToolImageTag: "quay.io/rhkp/incubator-kie-kogito-service-db-migration-postgresql:latest"
Comment on lines +33 to +34
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do not forget to replace with apache image

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed as soon as the image PR is approved and image is available, we will do the same.

# SonataFlow base builder image used in the internal Dockerfile to build workflow applications in preview profile
# Order of precedence is:
# 1. SonataFlowPlatform in the given namespace
Expand Down
2 changes: 2 additions & 0 deletions ...sonataflow-operator/config/manifests/bases/sonataflow-operator.clusterserviceversion.yaml
View file
Open in desktop
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the dbMigrationStrategy must be added to the DI and JS in this file too.

                      dbMigrationStrategy:
                        default: service
                        description: |-
                          DB Migration approach for service?
                          job: use job based approach
                          service: service itself shall migrate the db
                          none: no db migration needed

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@wmedvede Where are DI and JS in this file? Please advise.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rhkp your observation is good, and mine too 🤔

Rethinking here, It looks like we have a sort of outdated (and not necessarily outdated) specification of the CustomResourceDefinitions in this file... i.e., not all the fields were always added to the respective CRDs.

Some fields were added, and some other don't.

For example:
we have things like path: services.jobService.podTemplate but we don't have things like services.jobService.persistence

Which makes me think that not all the fields for the different CRDs owned by this operator where added to this file along the time. Question is, do we need them all added there? I think that not, see below.

Also, by looking at here: https://olm.operatorframework.io/docs/concepts/crds/clusterserviceversion I think the full specification of each CRD in that list is not needed (that information is not used for creating the respective CRDs), instead we could just only declare file the list of owned resources by this operator....

@ricardozanini is my reasoning correct?

Original file line number Diff line number Diff line change
Expand Up @@ -181,6 +181,8 @@ spec:
- description: Info generic information related to the Platform
displayName: info
path: info
- displayName: dbMigrationStatus
path: sonataFlowPlatformDBMigrationPhase
- description: Triggers list of triggers created for the SonataFlowPlatform
displayName: triggers
path: triggers
Expand Down
12 changes: 12 additions & 0 deletions packages/sonataflow-operator/config/rbac/role.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,18 @@ kind: ClusterRole
metadata:
name: manager-role
rules:
- apiGroups:
- batch
resources:
- jobs
verbs:
- create
- delete
- get
- list
- patch
- update
- watch
- apiGroups:
- monitoring.coreos.com
resources:
Expand Down
1 change: 1 addition & 0 deletions packages/sonataflow-operator/internal/controller/cfg/controllers_cfg.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,7 @@ type ControllersCfg struct {
JobsServiceEphemeralImageTag string `yaml:"jobsServiceEphemeralImageTag,omitempty"`
DataIndexPostgreSQLImageTag string `yaml:"dataIndexPostgreSQLImageTag,omitempty"`
DataIndexEphemeralImageTag string `yaml:"dataIndexEphemeralImageTag,omitempty"`
DbMigratorToolImageTag string `yaml:"dbMigratorToolImageTag,omitempty"`
SonataFlowBaseBuilderImageTag string `yaml:"sonataFlowBaseBuilderImageTag,omitempty"`
SonataFlowDevModeImageTag string `yaml:"sonataFlowDevModeImageTag,omitempty"`
BuilderConfigMapName string `yaml:"builderConfigMapName,omitempty"`
Expand Down
Loading
Loading