Bump istio installation

[aerosouund]

What this PR does / why we need it:

Bump Istio to the latest stable release

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #1324

Checklist

This checklist is not enforcing, but it's a reminder of items that could be relevant to every PR.
Approvers are expected to review this list.

• Design: A design document was considered and is present (link) or not required
• PR: The PR description is expressive enough and will help future contributors
• Code: Write code that humans can understand and Keep it simple
• Refactor: You have left the code cleaner than you found it (Boy Scout Rule)
• Upgrade: Impact of this change on upgrade flows was considered and addressed if required
• Testing: New code requires new unit tests. New features and bug fixes require at least on e2e test
• Documentation: A user-guide update was considered and is present (link) or not required. You want a user-guide update if it's a user facing feature / API change.
• Community: Announcement to kubevirt-dev was considered

[kubevirt-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign davidvossel for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[aerosouund]

/test check-provision-k8s-1.29

[mhenriks]

@aerosouund thanks for taking a crack at this! I am very keen to get this in. Do you need any help? The current version we are using is very old, and is not compatible with kubevirt/kubevirt#13422. The istio sidecar injection code is somehow stealing the runStrategy: Always bit from virt-tail. And since the istio container is not meant to always run, the virt-luancher pod never gets to running state.

cc @EdDev

[aerosouund]

@mhenriks
Thanks alot, I am facing a small challenge where after installation, checking the IstioOperator resource says that such a resource doesn't exist.
I am suspecting its because many things changed between the version we had and the version i am introducing 1.15 >> 1.24
I just need to check whats going on, but likely its not a big deal. I will ping you if i need any assistance!

[aerosouund]

On inspecting the cluster post running istioctl install -y (the operator init has been deprecated) it seems it the IstioOperator resource is no longer used

[vagrant@node01 ~]$ sudo kubectl --kubeconfig=/etc/kubernetes/admin.conf api-resources | grep istio
wasmplugins                                      extensions.istio.io/v1alpha1      true         WasmPlugin
destinationrules                    dr           networking.istio.io/v1            true         DestinationRule
envoyfilters                                     networking.istio.io/v1alpha3      true         EnvoyFilter
gateways                            gw           networking.istio.io/v1            true         Gateway
proxyconfigs                                     networking.istio.io/v1beta1       true         ProxyConfig
serviceentries                      se           networking.istio.io/v1            true         ServiceEntry
sidecars                                         networking.istio.io/v1            true         Sidecar
virtualservices                     vs           networking.istio.io/v1            true         VirtualService
workloadentries                     we           networking.istio.io/v1            true         WorkloadEntry
workloadgroups                      wg           networking.istio.io/v1            true         WorkloadGroup
authorizationpolicies               ap           security.istio.io/v1              true         AuthorizationPolicy
peerauthentications                 pa           security.istio.io/v1              true         PeerAuthentication
requestauthentications              ra           security.istio.io/v1              true         RequestAuthentication
telemetries                         telemetry    telemetry.istio.io/v1             true         Telemetry

With that said, the yaml files we use to install the operator will no longer work (istio-operator.cr.yaml & istio-operator-with-cnao.cr.yaml)
This signifies a huge migration from what we have, previously what you would have in the cluster is

[vagrant@node01 ~]$ sudo kubectl --kubeconfig=/etc/kubernetes/admin.conf get pods -A | grep istio
istio-operator   istio-operator-6c4fc4d784-2qqff            1/1     Running   0          2m50s
istio-system     istio-egressgateway-79c995f7cb-w4ppg       1/1     Running   0          88s
istio-system     istio-ingressgateway-775fdbc456-qq5cn      1/1     Running   0          88s
istio-system     istiod-5857496459-cnz5t                    1/1     Running   0          92s
kube-system      istio-cni-node-l99td                       1/1     Running   0          88s

After the upgrade, you only get

[vagrant@node01 ~]$ sudo kubectl --kubeconfig=/etc/kubernetes/admin.conf get pods -A | grep istio
istio-system   istio-ingressgateway-5f9df778cc-bl9sw      1/1     Running   0          33m
istio-system   istiod-69d6bb74c-z6fqk                     1/1     Running   0          34m

So we need to know how to get the same as what we had before in the previous version using 1.24. I might need help from the network team on this

cc: @oshoval @EdDev

[aerosouund]

Alternatively, we may not jump to 1.24. Maybe a lesser version that still behaves similar to 1.15 and has the things you want supported.
Needs a discussion for sure

[oshoval]

I am not familiar at all with Istio tbh, and currently on few other tasks,
if discussion will be still needed, i will try once i can, thanks

[aerosouund]

@oshoval
based on what changed across the deployments three questions need answered:

• is the egressgateway still needed or used ? has it's functionality been ported to any of the other components ?
• is the istiocni now being installed separately from the operator ? is yes, how ?
• would it be better to not skip to a very high version and upgrade to something intermediate? e.g: 1.18

[oshoval]

the first two need deeper understanding that i dont have now sorry,
if the 3rd helps advancing, lets go for that first thing, wdyt?
knowing where it stopped working, can also assist in debug latest sometimes

[aerosouund]

@mhenriks
whats the minimum compatible version that will contain the features you want ?

[mhenriks]

@mhenriks whats the minimum compatible version that will contain the features you want ?

1.20 is the oldest version that supports 1.29 (SideCar featuregate enabled) so that would be the minimum. But obviously latest would be best

https://istio.io/latest/docs/releases/supported-releases/#support-status-of-istio-releases

[ormergi]

* is the egressgateway still needed or used ? has it's functionality been ported to any of the other components ?

It seem we dont use in in e2e tests, we do use ingress-gateway though Istio API
https://github.com/kubevirt/kubevirt/blob/94d03645710c55dd35cf752bf6709bfb333518ad/tests/network/vmi_istio.go#L640
https://istio.io/latest/docs/setup/additional-setup/gateway/#deploying-a-gateway

* is the istiocni now being installed separately from the operator ? is yes, how ?

From what I remember Istio operator account for deploying its CNI, we have no dedicated scripting for doing that.
Check out https://istio.io/latest/docs/setup/additional-setup/cni/#installing-the-cni-node-agent

* would it be better to not skip to a very high version and upgrade to something intermediate? e.g: 1.18

Please note kubevirt e2e tests relays on the sidecar injection functionally.

I suggest to test this PR on kubevirt/kubevirt on sig-network lane so we can see where it fails and realize what our options.

[aerosouund]

@mhenriks @ormergi
Ok, based on this i will move the check for the IstioOperator resource (which is what's failing the tests here) and devise a way to install the CNI. Then we can run it against the sig network lane and see how it goes

[aerosouund]

@mhenriks
CI is looking good now, whats next ? is the code acceptable in its current state or do you have nay remarks ?

[akalenyu]

@mhenriks is out, but maybe @ormergi can help driver this further
BTW the PR title is a little bit misleading, since you're bumping the istio installation as a whole and not the cli tool

[ormergi]

@aerosouund please make sure istio bump pass all kubevirt/kubevirt CI lanes, specifically sig-network lanes.
Note that check-provision lanes run conformance tests only (subset of sig-network tests).
You can do that by posting a draft PR on kubevirt/kubevirt with this PR changes.

[aerosouund]

@ormergi This PR changes the gocli and the packages installed on the provider images in the provisioning phase. My assumption is that new versions for the gocli and providers need to be built and published on quay to make this testable on kv/kv since only the cluster-up part gets copied to kv/kv.
If those assumptions are false let me know and any suggestions you have on how we can test this

[akalenyu]

@ormergi This PR changes the gocli and the packages installed on the provider images in the provisioning phase. My assumption is that new versions for the gocli and providers need to be built and published on quay to make this testable on kv/kv since only the cluster-up part gets copied to kv/kv. If those assumptions are false let me know and any suggestions you have on how we can test this

It's actually possible to test prior to that - https://github.com/kubevirt/kubevirtci/blob/main/KUBEVIRTCI_LOCAL_TESTING.md

Ping me on k8s slack if you need help

[mhenriks]

/retest-required

[ormergi]

Hi, I managed to make istio-1.24 work on kubevirtci cluster, and verified istio tests are passing.
Including on kubevirt that is build with kubevirt/kubevirt#13422.

While troubleshooting the issues with 1.24 on kubevirtci, I saw istio-cni is not deployed as privileged , result VM creation failures.The privileged change is introduced in 1.24.
Istio 1.23 should work as expected, I suggest using this version.

In addition, IstioOperator API has been changed, see bellow config that worked for me:

kind: IstioOperator
metadata:
  namespace: istio-system
  name: istio-operator
spec:
  profile: demo
  components:
    cni:
      enabled: true
      namespace: kube-system
  values:
    global:
      jwtPolicy: first-party-jwt
    cni:
      chained: false
      cniBinDir: /opt/cni/bin
      cniConfDir: /etc/cni/multus/net.d
      privileged: true
      excludeNamespaces:
       - istio-system
       - kube-system
      logLevel: debug
    pilot:
      cni:
        enabled: true
        provider: "multus"
    sidecarInjectorWebhook:
      injectedAnnotations:
        k8s.v1.cni.cncf.io/networks: istio-cni

The missing configuration are:

• spec.values.pilot.cni
  This is a new API for triggering CNI deployment.
  The filed provider=multus acts like the known chained=false option (maybe in can be removed, need to verify).
• sidecarInjectorWebhook
  On istio >= 1.19, istio will append 'default/istio-cni' to the pod's k8s.v1.cni.cncf.io/networks is exist. Result in VMs failures.
  This cant work for kubevirtci env because we do not create the istio-cni NAD in the default namespace, but at the test namespace.
  In general it seems wrong because it means all pods in the cluster can connect to the mesh, becasuse all pods can connect to a NAD resied in default namespaces, no isolation allowed.

EDIT:
Related istio issues
istio/istio#54820
istio/istio#54815

[aerosouund]

@ormergi Thanks alot for your help on this

Istio 1.23 should work as expected, I suggest using this version.

Acknowledged, thanks.

This cant work for kubevirtci env because we do not create the istio-cni NAD in the default namespace, but at the test namespace.
In general it seems wrong because it means all pods in the cluster can connect to the mesh, becasuse all pods can connect to a NAD resied in default namespaces, no isolation allowed.

Not sure i quite follow through but in general i will test this on kubevirtCI and check if it produces the desired results

[kubevirt-bot]

@aerosouund: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
check-provision-k8s-1.29	ac32724	link	true	/test check-provision-k8s-1.29
check-up-kind-sriov	3a5b5f1	link	false	/test check-up-kind-sriov
check-provision-k8s-1.31-s390x	3a5b5f1	link	false	/test check-provision-k8s-1.31-s390x

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[kubevirt-bot]

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.