Some security advice about debug/pprof

[wenyurush]

In an internal security check, a medium-risk vulnerability was found in pushgetway, which was confirmed to be http://x.x.x.x:9091/debug/pprof related information.

According to the introduction may indeed generate some risks

https://www.farsightsecurity.com/blog/txt-record/go-remote-profiling-20161028/
http://mmcloughlin.com/posts/your-pprof-is-showing

Consider setting a switch to disable the relevant functionality when necessary

Thanks

[beorn7]

Indeed, I would recommend to never expose a pprof endpoint to untrusted networks.

However, it is even worse to expose the regular Pushgateway endpoints to untrusted networks. (The regular Pushgateway endpoints allow you to push arbitrary metrics to the Pushgateway and to delete all the pushed metrics, not to speak about inspecting the pushed metrics and the metrics of the PGW itself.)

A switch to disable pprof specifically would suggest that the remaining endpoints are somehow less critical to expose. But the opposite is the case.

[cturiel]

Hello Prometheus Pushgateway Team,

Our security team identified concerns with the pprof endpoint, relating to CVE-2019-11248. While we rigorously ensure that critical endpoints are not exposed on untrusted networks, the pprof exposure triggers security alerts in our audits.

Disabling "pprof" by default aligns with the principle of least privilege, reducing the attack surface without impacting functionality for users who do not require this endpoint.

I've proposed a feature in the PR #597 to disable the pprof endpoint by default, which can be enabled via a command-line flag.

Another option could be to expose pprof on another port.

Thanks for your work on Pushgateway.

Best regards,

[beorn7]

The line of argument above still holds. Exposing all the other endpoints of the PGW to untrusted networks is way worse than exposing the pprof endpoint. A feature to disable the pprof endpoint separately doesn't really make sense.

A security audit that flags exposure of the pprof endpoint while not flagging exposure of all the other PGW endpoints is just not doing a good job.

I would vote for keeping things as they are. Also, on a more general note, many other binaries in the Prometheus ecosystem expose the pprof endpoint by default, so a change of direction here should be done across the whole project. The suitable forum to discuss this is the developer mailing list. You also have the option of proposing a topic for the dev-summit.

[bwplotka]

Agree with pushgateway, but if we are creating a new convention on having a separate port for debug endpoints (metrics, pprof), we might as well do it for pushgateway, no? (assuming a new port is what we or users want in all other binaries)

[bwplotka]

Genrally I think we could consider making a pprof (or metrics) endpoints as a separate port (opt-in or opt-out). But if we want this for pushgateway, we might as well do it everywhere and start a new convention. As @beorn7 suggested, formal proposal of topic for the dev-summit would be amazing.

[beorn7]

Yes, we should definitely keep all the various prometheus/* binaries consistent with each other. That's why this should be discussed at the dev-summit to get cross project consensus.

[SuperQ]

I have yet to see any credible security vulnerability report on pprof being an actual exploit path.

However, I have been considering adding pprof initialization to be an exporter-toolkit feature. For making the configuration consistent, this would be the place to do it.

[beorn7]

Since there is substantial interest after all, I have proactively created a dev summit topic myself: https://docs.google.com/document/d/11LC3wJcVk00l8w5P3oLQ-m3Y37iom6INAMEu2ZAGIIE/edit?pli=1#bookmark=id.zbg4xs650bip

[SuperQ]

I don't think port is the question here. The question is enabling/disabling the entire pprof endpoint function.