feat: ORCA Format KV Cache Utilization in Inference Response Header#7839

Open

BenjaminBraunDev wants to merge 15 commits intotriton-inference-server:mainfrom BenjaminBraunDev:r24.10

+382

Commits on Jan 7, 2025

Add helper functions to pull metrics in HTTPAPIServer to pull metrics for use in HandleGenerate to add kv_utilization and max_token_capacity to the inference request response header.
BenjaminBraunDev
committed
Add logging, examples and more detailed comments, and move feature functionality to HTTPAPIServer::GenerateRequestClass::StartResponse() to extract metrics after inference request is processed for …
BenjaminBraunDev
committed

Commits on Jan 13, 2025

Commits on Jan 16, 2025

Merge branch 'main' into r24.10
BenjaminBraunDev
authored

Commits on Jan 22, 2025

Merge branch 'main' into r24.10
BenjaminBraunDev
authored

Commits on Jan 23, 2025

Commits on Jan 27, 2025

Merge branch 'main' into r24.10
BenjaminBraunDev
authored

Commits on Jan 28, 2025

Merge branch 'main' into r24.10
BenjaminBraunDev
authored

Commits on Jan 29, 2025

Merge branch 'main' into r24.10
BenjaminBraunDev
authored