This update represents a major rewrite of the package and introduces breaking changes. If you want to keep using the older version, you can download it using remotes::install_github("epiforecasts/[email protected]")
.
The update aims to make the package more modular and customisable and overall cleaner and easier to work with. In particular, we aimed to make the suggested workflows for evaluating forecasts more explicit and easier to follow (see visualisation below). To do that, we clarified input formats and made them consistent across all functions. We refactored many functions to S3-methods and introduced forecast
objects with separate classes for different types of forecasts. A new set of as_forecast_<type>()
functions was introduced to validate the data and convert inputs into a forecast
object (a data.table
with a forecast
class and an additional class corresponding to the forecast type (see below)). Another major update is the possibility for users to pass in their own scoring functions into score()
. We updated and improved all function documentation and added new vignettes to guide users through the package. Internally, we refactored the code, improved input checks, updated notifications (which now use the cli
package) and increased test coverage.
The most comprehensive documentation for the new package after the rewrite is the revised version
of our original scoringutils
paper.
See the NEWS file for a detailed overview of the changes.
What's Changed
- Replace use of ".Rda" by use of ".rds" where appropriate by @nikosbosse in #318
- Replace ..density.. by after_stat(density) by @nikosbosse in #319
- Create an S3 method for plot_avail_forecasts by @nikosbosse in #322
- Improvements to avail_forecasts() --> available_forecasts() by @nikosbosse in #321
- Add deprecation notes by @nikosbosse in #323
- Add documentation and make rounding for
correlation()
explicit by @nikosbosse in #320 - Rework scoring functions and their interface by @nikosbosse in #341
- Fix linting by @nikosbosse in #384
- Rework score() by @nikosbosse in #344
- Rework
summarise_scores()
andpairwise_comparison()
by @nikosbosse in #368 - Intermediate clean up by @nikosbosse in #375
- Update branch to include coverage functions by @nikosbosse in #394
- Update branch with older changes to manuscript by @nikosbosse in #440
- Rework quantile to interval format by @nikosbosse in #377
- First step towards reworking
score.scoringutils_quantile()
- adding coverage as a metric by @nikosbosse in #395 - Update
bias_quantile()
to work with vectors / matrices instead of data.table by @nikosbosse in #396 - Update functions to compute the absolute median by @nikosbosse in #419
- Rework score.scoringutils quantile()2 by @nikosbosse in #421
- Move tests around by @nikosbosse in #422
- Expand tests2 by @nikosbosse in #423
- Rework add coverage to work with raw forecasts by @nikosbosse in #426
- Simplify
score()
by @nikosbosse in #430 - Add separate functions for wis components by @nikosbosse in #397
- Fix set forecast unit by @nikosbosse in #437
- Add coverage deviation as a metric by @nikosbosse in #417
- Rework add coverage() by @nikosbosse in #390
- Rework quantile scores by @nikosbosse in #388
- Fix failing CI issues by deleting code remnants that shouldn't be there anymore by @nikosbosse in #478
- Fix failing CI issues by deleting code remnants that shouldn't be there anymore by @nikosbosse in #479
- Issue #480: Fix gh action by @nikosbosse in #483
- Fix small issues resulting from merging several updates into dev by @nikosbosse in #468
- Issue #405: expose
get_forecast_type()
to users by @nikosbosse in #466 - Issues #402: Expose
get_forecast_unit()
to users by @nikosbosse in #464 - Update input formats for binary and point forecasts by @nikosbosse in #460
- Fix pkgdown by @nikosbosse in #482
- Issue #500: Reduce messages in
bias_quantile()
by @nikosbosse in #501 - Issue #485: Fix linting by @nikosbosse in #509
- Issue #443 Drop interval functions by @nikosbosse in #525
- Issue #519: Fix rendering the Readme by @nikosbosse in #526
- Issue #275: Fix handling of scalar inputs in
logs_binary()
by @nikosbosse in #524 - Issue #446: Remove function
delete_columns()
by @nikosbosse in #529 - Issue #403: Rename
available_forecasts()
toget_forecast_counts()
by @nikosbosse in #511 - Issue #494: New workflow for creating and validating forecast objects by @nikosbosse in #531
- Issue #520: Rename forecast classes from
scoringutils_*
toforecast_*
by @nikosbosse in #533 - Issue #452: Add documentation for apply metrics by @nikosbosse in #470
- Issue 519: Fix failing render Readme action by @nikosbosse in #534
- Issue #404: Use
na.omit()
to removeNA
values before scoring by @nikosbosse in #465 - Issue #474: Make default scoring rules functions rather than stored data sets by @nikosbosse in #536
- Issue #448: Add back in examples for a few plots by @nikosbosse in #463
- Reduce the number of attributes used by
validate_forecast()
by @nikosbosse in #541 - Issue #436 Rename instances of "coverage" to either "interval coverage" or "quantile coverage" by @nikosbosse in #540
- Fix error message for
interval_coverage_quantile()
by @nikosbosse in #549 - Issue #552 Merge main into develop by @nikosbosse in #556
- Issue 553: interval_coverage_ testing improvements by @seabbs in #554
- Issue 551: Remove scoringutils metrics as using Metrics versions by @seabbs in #548
- Issue #535: Create pkgdown docs for stable and dev versions by @sbfnk in #550
- Issue 555: Rename interval_coverage_ family and remove sample version by @seabbs in #558
- Issue 560: Fix render_readme by @seabbs in #561
- Issue #564: add dependabot to keep GitHub Actions up to date by @sbfnk in #565
- Issue #566: don't clean upon pkgdown deployment. by @sbfnk in #567
- Issue #547: update package description in README by @sbfnk in #563
- Update render readme action in develop by @sbfnk in #576
- Issue #566: don't clean upon pkgdown deployment (this time on develop) by @sbfnk in #569
- Issue 557: Fix small numerical issue in sample_to_quantile() by @jhellewell14 in #570
- Develop into main by @sbfnk in #579
- Add current docs link to README by @seabbs in #582
- Issue #559: Ensure output of
add_coverage()
is an object of class forecast_quantile by @nikosbosse in #586 - Issues 520 and 484: expose function to get names of scores used in
score()
by @nikosbosse in #588 - PR #588 (issues #520 and #484): Update documentation for
get_score_names()
by @nikosbosse in #590 - Issue #475: rename output columns for pairwise comparisons by @sbfnk in #596
- Issue #597 - Creating a class
scores
for outputs ofscore()
by @nikosbosse in #606 - Related to Issue #545: Create function
is_forecast()
by @nikosbosse in #591 - Issue #580 validate output of
set_forecast_unit()
where appropriate by @nikosbosse in #613 - Bump docker/build-push-action from 2 to 5 by @dependabot in #574
- Bump n1hility/cancel-previous-runs from 2 to 3 by @dependabot in #575
- Bump docker/login-action from 1 to 3 by @dependabot in #573
- Bump actions/checkout from 2 to 4 by @dependabot in #572
- Bump docker/metadata-action from 2 to 5 by @dependabot in #571
- Issue #577: Rename
quantile
toquantile_level
by @nikosbosse in #612 - Issue #619: Update tests after data.table update by @nikosbosse in #618
- Issue #498: Implement a simple print function for forecast objects. by @toshiakiasakura in #592
- Unrelated changes - Roxygen update by @nikosbosse in #622
- Issue #593: Rename "range" to "interval_range" by @nikosbosse in #616
- Issue #627: implement attribute-preserving
[
by @sbfnk in #634 - Issue #629: Fix bug where output of pit() was a forecast object instead of a data.table by @nikosbosse in #630
- Issue 643: Fix partial matching by @jamesmbaazam in #644
- Issue #620: Allow print method to fail gracefully by @nikosbosse in #632
- Issue #400: Simplify
add_pairwise_comparison()
by @nikosbosse in #633 - Issue #585: allow users to specify columns and forecast unit in
as_forecast()
by @nikosbosse in #641 - Issue #649: Update snapshots to fix failing CI checks by @nikosbosse in #650
- Issue #645: Delete function
assert_equal_length()
by @nikosbosse in #654 - Issue #462: Delete function
plot_interval_ranges()
by @nikosbosse in #655 - Issue #617: Clean up
summarise_scores()
by @nikosbosse in #660 - Issue #656: Delete print.scoringutils_check by @nikosbosse in #657
- Issue #639: Remove
check_attribute_conflict()
by @nikosbosse in #658 - Issue #497: Rename argument
scores
tocoverage
in coverage plot functions by @nikosbosse in #664 - Issue #631: Create
get_coverage()
function by @nikosbosse in #665 - #659 Remove function
plot_predictions()
by @nikosbosse in #669 - Fix issue where
plot_interval_coverage()
drops columns by @nikosbosse in #678 - Issue #640: Update set forecast unit to error if any specified column is not found in the data by @nikosbosse in #679
- Issues #671 and 670: delete
check_quantiles()
andfilter_function_args()
by @nikosbosse in #680 - Use
{cli}
for condition signalling by @jamesmbaazam in #583 - Issue #507: Require users to call
as_forecast()
by @nikosbosse in #674 - Issue #653: Make
model
a required column by @nikosbosse in #676 - Add James A. as contributor by @jamesmbaazam in #690
- Fix readme by @nikosbosse in #691
- Delete unused functions by @nikosbosse in #698
- Move middle name to given to suppress roxygen warning by @jamesmbaazam in #694
- Update description and delete
scoringutils-package.R
by @nikosbosse in #695 - Replace
forecast_unit = NULL
withforecast_unit = get_forecast_unit(data)
by @nikosbosse in #697 - Add input checks to
log_shift()
by @nikosbosse in #696 - Update documentation of
log_shift()
by @nikosbosse in #699 - Issue #327 - Update package documentation by @nikosbosse in #692
- Issue #703: fix name of test file by @nikosbosse in #704
- Issue #707: Rename
score_names
tometrics
by @nikosbosse in #708 - Issue #383 - Remove
available_metrics()
by @nikosbosse in #706 - Issue #709: Rename
rules
tometrics
by @nikosbosse in #710 - Issue #562: Reorganise/rename files by @nikosbosse in #705
- #661 Issue - Add first version for new visualisation Vignette by @nikosbosse in #711
- reduce number of samples in pit examples by @nikosbosse in #715
- 637: Error in pairwise comparison by @sbfnk in #717
- Update docs by @nikosbosse in #720
- 651: fix pairwise plot by @sbfnk in #716
- Issue #638 - Rename
pit()
toget_pit()
by @nikosbosse in #724 - Issue #614 - update package imports by @nikosbosse in #723
- Issue #638 - Rename
correlation()
toget_correlations()
by @nikosbosse in #725 - Issue #714 - Update Readme by @nikosbosse in #719
- Rename pairwise comparisons by @nikosbosse in #722
- Issue #672 - Fix
get_correlations()
by @nikosbosse in #726 - Issue #635 - Remove warning about metrics that have previously been computed by @nikosbosse in #728
- Issue #677 - Warn users if they use an outdated format by @nikosbosse in #727
- Issue #245 - add additional input checks to
plot_correlations()
by @nikosbosse in #729 - Issue #355 - Add input checks to
plot_heatmap
by @nikosbosse in #733 - Issue #355 - Update input checks for
plot_forecast_counts()
by @nikosbosse in #735 - Issue #355 - Add input checks to
plot_wis()
by @nikosbosse in #732 - Issue #355 - Add input checks to coverage plots by @nikosbosse in #734
- Issue #636 - delete
plot_score_table()
and move code to deprecated visualisations vignette by @nikosbosse in #730 - Issue 450: get rid of library calls in tests/setup.R by @seabbs in #736
- Fix typo in import statement by @nikosbosse in #739
- Replace
by = NULL
withby = get_forecast_unit(data)
inget_forecast_counts()
by @nikosbosse in #741 - Issue #744 - Fix weird .internal.selfref error by making a copy in
check_number_per_forecast()
by @nikosbosse in #745 - Fix small typo in the readme by @nikosbosse in #760
- Issue #750 - Introduce a
forecast_type
argument invalidate_forecast()
by @nikosbosse in #751 - Issue #742 - simplify
pit_sample()
by @nikosbosse in #743 - Issue #447 - Add input checks to functions by @nikosbosse in #748
- Issue #425 - Updating package documentation by @nikosbosse in #747
- Issue #447 - Add even more input checks by @nikosbosse in #756
- Issue #666 - Revise manuscript by @nikosbosse in #544
- Issue #765 - Delete old overviews of the default metrics by @nikosbosse in #764
- Issue #761 - Update input checks and docs for
quantile_score()
by @nikosbosse in #768 - Issue #755 - Make
sample_to_quantile()
require aforecast
object by @nikosbosse in #770 - change
data
toforecast
inscore()
and update related tests by @nikosbosse in #772 - Issue #594 - Update documentation for
get_pairwise_comparisons()
by @nikosbosse in #737 - Issue #693 - Remove
forecast_unit
argument fromget_duplicate_forecasts()
by @nikosbosse in #774 - Issue #738 - Rename continuous and discrete examples by @nikosbosse in #775
- Issue #447 - Add and simplify more input checks by @nikosbosse in #753
- Update Rd docs and correct vignette title by @nikosbosse in #777
- Issue #754 - Drop function
merge_and_pred_obs()
by @nikosbosse in #776 - Cosmetic changes and spell-checking by @nikosbosse in #779
- Issue #688 - Reorganise
validate_forecast()
by @nikosbosse in #778 - Correct docs for brier score by @nikosbosse in #783
- Issue #309 - fix error in bias quantile documentation by @nikosbosse in #782
- Issue #758 - Update metrics vignette by @nikosbosse in #762
- Issue #687 - Reorganise NEWS file before release by @nikosbosse in #781
- Issue #310 - Improve imputation of median in
bias_quantile()
by @nikosbosse in #786 - Issue 431: Stop passing arguments to underlying scoring rules in score and provide a better helper. by @seabbs in #787
- Add manuscript to git by @nikosbosse in #795
- Ensure that summarise_scores() works with a data.frame or similar by @nikosbosse in #805
- Applying code suggestions from code review by @nikosbosse in #806
- Issue #771 - Make quantile_to_interval() an internal function by @nikosbosse in #794
- Issue #801 - Improve apply metrics and
run_safely()
by @nikosbosse in #803 - Issue ##438 - Add tests to check existing data.table functions work with data.frames by @nikosbosse in #799
- Issue #784 - Add an entry for the scoringutils paper on the pkgdown site by @nikosbosse in #797
- Issue #444 - Remove
interval_long_to_quantile
(previouslyrange_long_to_quantile
) by @nikosbosse in #798 - Issues #502 and #587- Add additional tests by @nikosbosse in #802
- Partially merge changes suggested by code review in #791 by @nikosbosse in #819
- Review scoringutils 2.0.0 by @Bisaloo in #791
- Issue #809 - Replace is.logical by isTRUE by @nikosbosse in #815
- Clean a few minor things by @nikosbosse in #814
- Issue #828 - Allow
score()
to succeed even if some scores can't be computed by @nikosbosse in #829 - remove across argument from
summarise_scores
by @nikosbosse in #831 - Issue #824 - Remove unnecessary files by @nikosbosse in #830
- Issue #812 - mention hubVis package in vignette by @nikosbosse in #833
- Issue #818 - Make assert_forecast_generic internal by @nikosbosse in #835
- Issue #759 - Remove scoringutils vignette by @nikosbosse in #836
- Issue #821 - Remove digits argument from get_correlations by @nikosbosse in #834
- Issue #628 - Improve duplicate check by @nikosbosse in #841
- Require R4.0 by @nikosbosse in #843
- Issue #817 - Create forceast super class by @nikosbosse in #813
- Bump docker/build-push-action from 5 to 6 by @dependabot in #845
- Issue #838 - small fix to input checks by @nikosbosse in #850
- Issue #855 - render readme as PR by @nikosbosse in #857
- Use ASCII hypen by @MichaelChirico in #863
- Issue #861 - fix failing render readme by @nikosbosse in #865
- ns-qualify as.data.table() for R CMD check by @MichaelChirico in #864
- Issue #847 - Update funding statements by @nikosbosse in #858
- Issue #848 - Split
as_forecast()
into separate functions for the different forecast types by @nikosbosse in #849 - issue #827 - Improve error handling for wis() by @nikosbosse in #859
- Update README by @github-actions in #866
- More robust message wording test by @MichaelChirico in #862
- Updates to crps_split PR by @nikosbosse in #869
- 853: add CRPS decomposition by @sbfnk in #854
- Fix inheritance order by @Bisaloo in #871
- #844 - Add pictures to function documentation by @nikosbosse in #879
- Issue #530 - Clarify workflow in examples by @nikosbosse in #877
- Issue #820 - Replace customise_metric with
purrr::partial
by @nikosbosse in #874 - Issue #851 - Create S3 methods for converting to point and quantile forecasts by @nikosbosse in #876
- Issue #800 - Add formula and reference to WIS documentation by @nikosbosse in #883
- Issue #878 - Fix/Implement quantile score for a vector input by @nikosbosse in #882
- Issue #481: Update pkgdown structure by @nikosbosse in #886
- Issue #604 - Add support for nominal forecasts by @nikosbosse in #837
- Update README by @github-actions in #891
- Create custom [.forecast() method by @Bisaloo in https://github.com//pull/884
- Bump peter-evans/create-pull-request from 6 to 7 by @dependabot in #899
- Use stricter check to bypass validation of x[] by @Bisaloo in #897
- Issue #887 - Make
get_forecast_type()
an internal function by @nikosbosse in #893 - Issue #832 - Make example data pre-validated by @nikosbosse in #901
- Increase test coverage by @nikosbosse in #906
- Issue #902 - Make
set_forecast_unit()
internal by @nikosbosse in #905 - Issue #900: Remove observed and predicted columns from output for point and binary forecasts by @nikosbosse in #904
- Issue #832 - Make default metrics function S3 by @nikosbosse in #903
- 359: convert get_pit() to S3 by @sbfnk in #910
- Improve function documentation for point forecasts by @nikosbosse in #911
- 359: Check PIT by @sbfnk in #916
- Update suggests by @nikosbosse in #917
- fix names by @nikosbosse in #920
- #896 Update documentation of log score by @nikosbosse in #918
- 908: Flexible model column by @sbfnk in #915
- Clean up scripts used for the old manuscript by @nikosbosse in #936
- Issue #925 - Remove function
interval_coverage_deviation()
by @nikosbosse in #928 - #885 - Delete
validate_forecast()
by @nikosbosse in #921 - Update README by @github-actions in #937
- #931 - Fix data.table issue caused by
attr()
by @nikosbosse in #932 - Improve error handling for
get_forecast_counts()
and streamline some tests by @nikosbosse in #923 - Issue #502 - Add tests and Improve error handling by @nikosbosse in #924
- #746 - Reorganise files by @nikosbosse in #940
- Handling case in WIS where quantile levels don't form valid intervals by @nikosbosse in #943
- Update startup message and rename file to make clear what's inside by @nikosbosse in #945
- Update manuscript by @nikosbosse in #946
- 913: Functions for PIT histograms by @sbfnk in #949
- Fix some warnings that appeared in tests by @nikosbosse in #954
- update
return
with the newreturns
by @nikosbosse in #957 - Fix notes related to CRAN checks by @nikosbosse in #958
- Update README by @github-actions in #959
- Issuel #952 - Fix get_forecast_type by @nikosbosse in #953
- Split out
as_forecast()
by @nikosbosse in #955 - Remove function that resulted in an error when called from within print.data.table() by @nikosbosse in #956
- Cran updates by @nikosbosse in #960
- Update README by @github-actions in #963
- Remove mentions of the getting started vignette which doesn't exist a… by @nikosbosse in #964
- Update README by @github-actions in #965
New Contributors
- @jhellewell14 made their first contribution in #570
- @dependabot made their first contribution in #574
- @toshiakiasakura made their first contribution in #592
- @jamesmbaazam made their first contribution in #644
- @MichaelChirico made their first contribution in #863
- @github-actions made their first contribution in #866
Full Changelog: v1.2.2...v2.0.0