Reading Rubrix

When reading a research paper, it is possible extract certain information from a paper

About the Paper

1. Motivational statements	reports or challenge statements or lists of open issues that prompt an analysis;
2. Hypotheses	Expected effects in some area;.
3. Checklists	Used to design the analysis (see also, the Checklist Manifesto] ;.
4. Related Work	Comprehensive, annotated, and insightful (e.g. showing the development or open areas in a field);.
5. Study instruments	e.g. surveys interview scripts, etc;.
6. Statistical tests	Mathematical tools to analyze results (along with some notes explaining why or when this test is necessary);.
7. Commentary	About the scripts used in the analysis;.
8. Informative visualizations	e.g. Sparklines http://www.edwardtufte.com/bboard/q-and-a-fetch-msg?msgid=0001OR .
9. Baseline results	Results against which new work can be compared;.
10. Sampling procedures	e.g. ``how did you choose the projects you studied?'';.
11. Patterns	describing best practices for performing this kind of analysis; .
Anti-patterns	describing cautionary tales of ``gotchas'' to avoid when doing this kind of work;
12. Negative results	Anti-patterns, backed up by empirical results;.
13. Tutorial materials	Guides to help newcomers become proficient in the area. Some of these tutorial materials may be generated by the researcher and others may be collected from other sources..
14. New results	Guidance on how to best handle future problems..
15. Future work:	Based on the results, speculations about open issues of future issues that might become the motivation for the next round of research.

The follow are usually too large to be included in a paper. However, a paper can point to some external resource that includes...

16. Data	Used in an analysis; either raw from a project; or some derived product.
17. Scripts	used to perform the analysis (the main analysis or the subsequent statistical tests or visualizations; e.g. the Python Sparklines generator or code for a fast a12 test. Scripts can also implement some of the patterns identified by the paper.
18. Sample models	Can generate exemplar data; or which offer an executable form of current hypotheses. Or, these models could be a set of standard problems everyone shares (e.g. the verification comminity and optimization community have libraries of standard models (or models ported from commercial apps) that they all use to baseline their tools)
29. Delivery tools	Things that let other people automatically rerun the analysis; e.g. + Config management files that can + build the system/ paper from raw material and/or + update the relevant files using some package manager + Virtual machines containing all the above scripts, data, etc, pre-configured such that a newcomer can automatically run the old analysis.