Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Function to get ordered levels of output_type_ids for pmf or categorical output types #153

Open
elray1 opened this issue Jun 27, 2024 · 0 comments
Labels
feature a feature request or enhancement

Comments

@elray1
Copy link
Contributor

elray1 commented Jun 27, 2024

For the cdf output type as well as the pmf output type for targets with a target_type of ordinal, the order of the output_type_ids matters for validation, plotting and evaluation:

  • For validation of cdf types, we need to ensure that the probabilities are non-decreasing across the category levels; this requires knowing the correct order of the category levels. See this issue.
  • For plotting, we will probably want to arrange the categories in the right order
  • For evaluation, some scores depend on getting the categories in the right order (e.g., this is required to calculate RPS for pmf output type)

To support those use cases, I propose adding a function here that extracts the output_type_id values that were specified in the tasks.json file, in the order they were specified there (so that we are taking the order they are given in the tasks.json file as a specification of the order of those levels.). This function probably would need to take a round_id and enough information to look up the category orders -- probably output_type and some set of task id variables?

@elray1 elray1 added the feature a feature request or enhancement label Jun 27, 2024
@nickreich nickreich moved this from Todo to Up Next in hubverse Development overview Jul 17, 2024
@zkamvar zkamvar assigned zkamvar and unassigned zkamvar Aug 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature a feature request or enhancement
Projects
Status: Todo
Development

No branches or pull requests

2 participants