Clarify interactions between TorchServe and KServe

[kimminw00]

📚 The doc issue

When deploying PyTorch models using the pytorch/torchserve-kfs image with Kserve, I found it challenging to understand the architecture and how different processes interact with each other. Specifically, I would like to know which processes run in which pods and how resources are allocated. To optimize for large traffic volumes, it's crucial to understand how resources are allocated to each process.

As I understand, TorchServe uses Netty-based HTTP/gRPC servers, while Kserve uses Tornado-based HTTP/gRPC servers. However, when deploying with pytorch/torchserve-kfs image, it's unclear what process runs where.

Reference
https://kserve.github.io/website/master/modelserving/v1beta1/torchserve/

Suggest a potential alternative/fix

If possible, providing a high-level diagram or explanation of how the different components interact would be incredibly helpful.

[cjidboon94]

Hey @kimminw00 as far as my understanding goes the following happens in the kserve-container:
Two processes get started:

• a thin frontend FastAPI-based Kserve Modelserver
• a backend Torchserve server which manages your models.

On startup, Kserve frontend server waits until torchserve is ready with loading all your models (checks status by pinging it every few seconds
Once the model is loaded and kserve got a positive response from the torchserve it marks and registers the model as ready to be used and I believe the health check then passes and the pod is also considered ready.

When requests come in, they go first through the Kserve modelserver, get redirected to the torchserve server ,returned to the Kserve server which then finally returns it to the client.

Graphically this would be:

Request ->   KServe -> Torchserve
Response <- Kserve <- Torchserve