Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create model selection SOP #1

Open
4 tasks
dfsnow opened this issue Jul 10, 2023 · 1 comment
Open
4 tasks

Create model selection SOP #1

dfsnow opened this issue Jul 10, 2023 · 1 comment
Assignees
Labels
sop Standard Operating Procedures

Comments

@dfsnow
Copy link
Member

dfsnow commented Jul 10, 2023

We should write a Standard Operating Procedure (SOP) codifying how and why we select final valuation model runs. This is mostly about formalizing and documenting the best practices and making sure that we're implementing them internally.

  • Collect examples of similar SOPs from other departments/companies
  • Collect best practices re: model selection, see Tidymodels docs, Max Kuhn, other predictive modeling resources
  • Seek feedback from Valuations on any proposed changes
  • Publish the SOP to the wiki, with link from README
@dfsnow dfsnow added the new label Jul 10, 2023
@dfsnow dfsnow self-assigned this Jul 17, 2023
@dfsnow dfsnow added sop Standard Operating Procedures and removed new labels Jul 20, 2023
@dfsnow
Copy link
Member Author

dfsnow commented Jul 20, 2023

We should also reconsider using our current method for bootstrapped CI calculations. We rarely use the CIs now and I'm not confident that the original implementation works as intended. See https://stats.stackexchange.com/questions/433866/bootstrapped-confidence-intervals-for-performance-metrics-of-predictive-models.

@dfsnow dfsnow removed their assignment Aug 3, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
sop Standard Operating Procedures
Projects
None yet
Development

No branches or pull requests

2 participants