Using ADM Datamart Object - Model Performance Stats

[sushds]

Hi team,

I have been able to convert my model exports into a datamart object.

I notice that there are quite a few functions available to explore but I am not sure if there is something specifically on getting metrics on model performance other than AUC such as: precision, recall, f-score and information about sample size. In my experience I have used this metrics to evaluate performance of a classification model.

Could you direct me where I could find these metrics or alternatively calculate these?

I would also like to request to include these stats in the ADM health check report.

Thank you!

[sushds]

Hi @operdeck / @StijnKas, not sure if someone from team received notification for this question so tagging both of you!

[StijnKas]

Hi @sushantkhare, apologies for the delayed response.

You're right, we don't provide these methods out of the box in pdstools, as we consider AUC to be the most appropriate metric for evaluating our adaptive models. Of course it's definitely possible to calculate them, and we use them internally to recalculate AUC here: https://github.com/pegasystems/pega-datascientist-tools/blob/master/python/pdstools/utils/cdh_utils.py#L302. You can reference this implementation if you want to derive your own functions.

In the R health check we've already included the PR (Precision/Recall) AUC as well, and we're considering adding them to the python health check too. Is there a particular reason you mention precision/recall/f-score should be included in the health check? As of right now we don't have plans to include them, but if there's a valid use case for it we can always reconsider that!

I hope that helps, else let us know!

[sushds]

Hi @StijnKas, thanks for the response!

To steer on the model performance, I have always used metrics such as Precision, Recall and F-Score in conjunction with AUC score. I think these metrics are important to consider if there is class imbalance. Precision, recall, and F-score provide more specific information regarding how well the model performs in correctly identifying a specific class. AUC, while helpful in measuring the model's ability to distinguish between classes, does not provide class-specific details.

Since they are already included in R and my reasoning above I hope this provides a valid use case to include them in the python scripts as well. :)

Ideally these are the type of stats I would like to see to compare model performance when we perform experimentation:

P. S. Don't ask me what McNemar's test and Kappa are - never used those two😃