Repository Structure

This repository contains the data and scripts to replicate the results of the paper "Quality of Automated Program Repair on Real-World Defects". It also contains the pre-computed results and experiment logs for easy reference. Following describes the repository structure.

• The src directory contains the source code used for:
  • repair-technique-execution: This directory contains the scrips used to execute the repair techniques using JaRFly framework.
  • test-generation-and-sampling: This directory contains the scripts used for generating evaluation test suites using EvoSuite and sampling test suites based on statement coverage and number of failing tests.
  • patch-quality-assessment: This directory contains the scripts used to assess the quality of the patches produced by repair techniques using held-out evaluation test suite.
  • statistical-tests-execution: This directory contains, for each research question, the scripts used to run the corresponding statistical tests and generate plots.
• The results directory contains the result files organized in terms of the research questions described in the paper.
  • RQ1(Do G&V techniques produce patches for real-world defects?): This directory contains patches (patches produced by repair techniques) and statistical-tests (results showing if the repair techniques can patch real-world defects) sub-directories.
  • RQ2(How often and how much do the patches produced by G&V techniques overfit to the developer-written test suite and fail to generalize to the evaluation test suite, and thus ultimately to the program specification?): This directory contains patches (patches produced by repair techniques), patch-quality-assessment (quality assessment of patches), and statistical-tests (results showing the patch overfitting) sub-directories.
  • RQ3(How do the coverage and size of the test suite used to produce the patch affect patch quality?): This directory contains sampled-tests-coverage (developer-written tests sampled based on statement coverage), patches (patches produced by repair techniques), patch-quality-assessment (quality assessment of patches), and statistical-tests (results showing the affect of test suite coverage and test suite size on patch quality) sub-directories.
  • RQ4(How does the number of tests that a buggy program fails affect the degree to which the generated patches overfit?): This directory contains sampled-tests-failing-test-count (developer-written tests sampled based on failing test count), patches (patches produced by repair techniques), patch-quality-assessment (quality assessment of patches), and statistical-tests (results showing the effect of failing test count on patch quality) sub-directories.
  • RQ5(How does the test suite provenance (whether it is written by developers or generated automatically) influence patch quality?): This directory contains patches (patches produced by repair techniques using EvoSuite-generated tests to guide the repair), patch-quality-assessment (quality assessment of patches), and statistical-tests (results showing the effect of test provenance on patch overfitting) sub-directories.
  • RQ6(Can overfitting be mitigated by exploiting randomness in the repair process? Do different random seeds overfit in different ways?): This directory contains patch-quality-assessment (quality assessment of patches produced using developer-written tests, same as RQ2), test-execution-results(quality assessment combined patches (n-version)) and statistical-tests (results comparing the quality of n-version patches with original patches) sub-directories.
• The test_suites directory contains the independely generated held-out test suites using EvoSuite version 3 (Evosuite103Budget1800) optimizing the branch coverage criterion and using EvoSuite version 6 (Evosuite106Budget1800)optimizing the line coverage criterion.
• The experiment_logs directory contains the logs generated while executing the repair techniques --- GenProg, Par, and TRPAutoRepair to address each of the research questions listed above. The logs are organized in terms of research questions.