Add kubernetes deployment for GenAIComps

[yongfengdu]

Description

The summary of the proposed changes as long as the relevant motivation and context.

Issues

List the issue or RFC link this PR is working on. If there is no such link, please mark it as n/a.

Type of change

List the type of change like below. Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds new functionality)
• Breaking change (fix or feature that would break existing design and interface)
• Others (enhancement, documentation, validation, etc.)

Dependencies

List the newly introduced 3rd party dependency if exists.

Tests

Describe the tests that you ran to verify your changes.

[mkbhanda]

Some questions ..

Any reason why no failureThreshold on the readinessProbe for CPU and gaudi values?

Why do we not mention any image for CPU .. is it in some common file. Likewise all the min/max token lengths.

What about model id with Gaudi?

Thank you @yongfengdu for this PR.

[yongfengdu]

Some questions ..

Any reason why no failureThreshold on the readinessProbe for CPU and gaudi values?
These values were tuned by @eero-t and I just borrowed them.
The default failureThreshold is 3.
For livenessProbe failure, the pod will be killed/restarted by kubelet, I think that's why the failureThreshold was set to a bigger value.

Why do we not mention any image for CPU .. is it in some common file. Likewise all the min/max token lengths.
There are default values in values.yaml, which is included in the helm chart. But it might be a good idea to mention them in cpu-values.yaml and gaudi-values.yaml, even if they are the same as default values.yaml.
I'll refine the *-values.yaml file.
What about model id with Gaudi?

Thank you @yongfengdu for this PR.

[yongfengdu]

Here is the link for default values.
https://github.com/opea-project/GenAIInfra/blob/main/helm-charts/common/tgi/values.yaml
It's better not set duplicate VARs to reduce maintenance effort.
For example the TGI image's tag:
tag: "2.4.0-intel-cpu"
It's a value need frequently change, we won't have it in multiple places.

Some questions ..

Any reason why no failureThreshold on the readinessProbe for CPU and gaudi values?

Why do we not mention any image for CPU .. is it in some common file. Likewise all the min/max token lengths.

What about model id with Gaudi?

Thank you @yongfengdu for this PR.

[eero-t]

Any reason why no failureThreshold on the readinessProbe for CPU and gaudi values?

These values were tuned by @eero-t and I just borrowed them.

I was concentrating more on when things actually start working, rather than when they start failing, so it's a rather rough value.

The default failureThreshold is 3.

They could be explicitly mentioned also for readinessProbes.

For Gaudi, I think the readiness probe failureThreshold could be smaller, but I haven't tested it.

For CPU it's definitely better to keep it >1 because perf on CPU is so unpredictable (because underlying HW can differ and pods do not have fine-tuned resource requests / limits).

For livenessProbe failure, the pod will be killed/restarted by kubelet, I think that's why the failureThreshold was set to a bigger value.

I've never seen OPEA pods deadlock, especially in a way that would be solved by restarting the pod. I.e. liveness probe restarts just make things worse, as service may then never reach ready/live state => IMHO liveness probes are just harmful, and could be removed.

[poussa]

Why do we have kubernetes/deployment in the pathname? It should be either kubernetes or deployment, not both. Or what is the logic here?

[yongfengdu]

Why do we have kubernetes/deployment in the pathname? It should be either kubernetes or deployment, not both. Or what is the logic here?

It's "deployment/kubernetes"
There is also a docker_compose directory under deployment(deployment/docker_compose), providing the instructions for docker environment.
See
https://github.com/opea-project/GenAIComps/tree/main/comps/3rd_parties/tgi/deployment

[lianhao]

for dataprep and retriever, the current value files will be obsolete in a couple of days.(new PR in infra repo will be submitted next week). Should we remove them from this PR?