kubectl cost predict -f test.yaml Not working

[HibAwad]

I have the access to the Kubecost page on the localhost 9091.
When running the command 'kubectl cost predict -f test.yaml' it is giving an error:

Error: prediction query failed: failed to port forward query: received non-200 status code 404 and data: 404 page not found

[cliffcolvin]

@HibAwad thank you so much for reaching out on your issues, and I appreciate in advance your patience with us on this. I have asked one of our engineers to take a look at this one and we'll reach back out as soon as we can to either gather some more information or discuss resolution steps.

[nickcurie]

Hi @HibAwad would you mind running that command again with the --log-level debug flag and sending the output here, so that I could take a deeper look into what might be going wrong here? Thank you!

[HibAwad]

Hello @nickcurie I tried it:

[HibAwad]

kubectl cost predict -f testspecs.yaml --log-level debug
2024-09-06T12:47:09.399850465+02:00 DBG ServiceName set to: kubecost-cost-analyzer
2024-09-06T12:47:09.399884205+02:00 DBG KubecostNamespace set to: kubecost
2024-09-06T12:47:09.410731409+02:00 DBG checking readiness of 'kubecost-cost-analyzer-7dd99d475f-89v6h'
2024-09-06T12:47:09.410755607+02:00 DBG selected pod to forward: kubecost-cost-analyzer-7dd99d475f-89v6h
2024-09-06T12:47:09.424228064+02:00 DBG Port-forward set up at: http://localhost:46085
2024-09-06T12:47:09.424283893+02:00 DBG Executing GET to: http://localhost:46085/model/clusterInfo
2024-09-06T12:47:09.439839635+02:00 DBG Cluster ID for query set to: cluster-one
2024-09-06T12:47:09.439879839+02:00 DBG Executing POST to: http://localhost:46085/model/prediction/speccost?clusterID=cluster-one&defaultNamespace=default&noUsage=false&windowAvgUsage=2d&windowResourceCost=7d+offset+48h
Error: Failed querying the speccost API. This API requires a version of Kubecost >= 1.101, which may be why this query failed. If running Kubecost v1.100, you can downgrade kubectl cost to v0.4 for old-style prediction. Error: failed to port forward query: received non-200 status code 500 and data: Failed to predict cost impact of specs: AllocationSet is empty

[HibAwad]

I tried also to downgrade the version of kubectl cost to V0.4 and I got this error:

kubectl cost predict -f /tmp/testspecs.yaml
Error: prediction query failed: failed to port forward query: received non-200 status code 404 and data: 404 page not found

kubectl cost predict -f testspecs.yaml --log-level debug
2024-09-06T14:16:44.299409325+02:00 DBG ServiceName set to: kubecost-cost-analyzer
2024-09-06T14:16:44.299445172+02:00 DBG KubecostNamespace set to: kubecost
2024-09-06T14:16:44.310128794+02:00 DBG checking readiness of 'kubecost-cost-analyzer-644f5cc4cd-g9q5r'
2024-09-06T14:16:44.310178445+02:00 DBG selected pod to forward: kubecost-cost-analyzer-644f5cc4cd-g9q5r
2024-09-06T14:16:44.330726091+02:00 DBG Port-forward set up at: http://localhost:38987
2024-09-06T14:16:44.334065628+02:00 DBG Executing GET to: http://localhost:38987/model/prediction/resourcecostdiff?clusterID=&controllerKind=deployment&controllerName=kubecost-cost-analyzer&controllerNamespace=kubecost&requestedCPU=6m&requestedGPU=0&requestedMemory=6Mi&window=2d
Error: prediction query failed: failed to port forward query: received non-200 status code 404 and data: 404 page not found

[HibAwad]

curl http://localhost:9090/model/predection
404 page not found

[cliffcolvin]

@HibAwad I think we have a couple of issues going on.
I believe this to be the issue in versions. Which version of kubecost and which version of kubectl-cost are you using? I'd recommend 2.3.5 as of today on kubecost and v0.6.3 on kubectl-cost.
I believe we removed /model/prediction in our modernization efforts when moving toward kubecost 2.0. Kubectl-cost for some time had been moved off of /model/prediction, resourcediff and over to speccost endpoint.

If you can help me get your version information I'll try and reproduce what you are seeing so we can verify this is in fact the culprit before you start trying to upgrade anything.

In an effort at thoroughness in my search to help can you also upload the yaml you are using?

[HibAwad]

I am trying this :
read -r -d '' DEF << EndOfMessage
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
namespace: default
labels:
app: nginx
spec:
replicas: 3
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.14.2
resources:
requests:
cpu: "3"
memory: "2Gi"
EndOfMessage
echo "$DEF" | kubectl cost predict -f -

My kubecost version is 2.3.5
kubectl-cost is V0.4 and I got the error of Page not found

I tried already V0/6/3 (latest) and I got the error AllocationSet is empty

I'll retry to use the latest version of kubectl-cost V0.6.3

[HibAwad]

Kubecost V2.3.5
kubectl-cost v0.6.3

[HibAwad]

I also notified these warnings on my localhost:9090

I don't know if they are related

[AjayTripathy]

Hi @HibAwad you'll need to make sure cadvisor is being scraped to get meaningful predictions. Looks like it isn't here. Can you try using the default prometheus installed with Kubecost if you aren't already?

[HibAwad]

Hello @AjayTripathy I verified via :
kubectl port-forward svc/kubecost-prometheus-server -n kubecost 9090:80

and I have it

[caiyuanji]

I had the same problem.
helm kubecost/cost-analyzer-2.4.0
plugin cost v0.6.3

[HibAwad]

Hello, I am closing this issue because kubecost is now working for me.
If anyone has the error of allocation is empty, you can try to wait for 3 days and after that test the prediction again while ensuring that the pods are all running and ready.