Skip to content

nus-apr/defects4j-instrumented

BranchesTags

Repository files navigation

defects4j-instrumented

This repository includes the instrumented defects4j subjects. We manually transformed the information from the bug report into an oracle, which is added to the buggy version of each subject.

Each modification is tagged as: D4J_<project>_<bugID>_BUGGY_VERSION_INSTRUMENTED

The zipped git archives are included in instrumented-archives.

The diff files are included in instrumented-diffs.

Note: For now, we will only have direct modifications of the buggy source code. Later, we want to introduce an automated mechanism that (given the source code modification) directly instruments the Java bytecode.

Current general issues

  • not all test cases call the method mentioned in the bug report; they call the underlying buggy method
  • with the additional specificaton from the bug report, other test cases can also fail. They should be fixed with the correct bug though. E.g., Lang-7, a test case interpretes both (the incorrect and the correct) behavior a suitable way so that the test cases passes; but with our instrumentation it fails because we throw a RuntimeException. Correct: throw NumberFormatException. Buggy: return null. The other test case handles both but not our RuntimeException.

Methodology: Subject Selection

  • we start with subjects, for which ARJA can generated plausible patch
  • ARJA
    • considers 224 bugs from Defects4J
    • plausible patches: 59
    • correct patches: 18

Methodology: Instrumentation Writing

  1. TODO

Example: Lang-19

diff --git a/src/main/java/org/apache/commons/lang3/StringEscapeUtils.java b/src/main/java/org/apache/commons/lang3/StringEscapeUtils.java
index d84aae58..f5757eac 100644
--- a/src/main/java/org/apache/commons/lang3/StringEscapeUtils.java
+++ b/src/main/java/org/apache/commons/lang3/StringEscapeUtils.java
@@ -465,7 +465,23 @@ public class StringEscapeUtils {
      * @since 3.0
      */
     public static final String unescapeHtml4(String input) {
-        return UNESCAPE_HTML4.translate(input);
+       if (Boolean.parseBoolean(System.getProperty("defects4j.instrumentation.enabled"))) {
+            try {
+                // Original Code START
+                return UNESCAPE_HTML4.translate(input);
+                // Original Code END
+            } catch (StringIndexOutOfBoundsException e) {
+                if (input.contains("&")) {
+                    throw new RuntimeException("[Defects4J_BugReport_Violation]");
+                } else {
+                    throw e;
+                }
+            }
+       } else {
+            // Original Code START
+            return UNESCAPE_HTML4.translate(input);
+            // Original Code END
+       }
     }

Additionally, we insert the comment // defects4j.instrumentation at the end of each line of the oracle's instrumentation. This is done so that an automated tool can detect the oracle and, e.g., avoid a repair engine removing the oracle during the patch generation.

Instrumented Subjects Overview (43 + 3)

ARJA reported that it can generate plausible patches for 59 subjects. From these 59, we instrumented in total 43 subjects. From these 43,

  • 29 subjects are covered by ARJA with plausible but incorrect patches, and
  • 14 subjects are covered by ARJA with correct patches.

For 20 subjects we were not able to write an oracle (see below).

Additionally, we instrumented 2 subjects, for which ARJA cannot generate any plausible patch (see below).

commons-lang (15 subjects)

Lang-7 (ARJA plausible but incorrect)
Lang-16 (ARJA plausible but incorrect)
Lang-20 (ARJA correct)
Lang-22 (ARJA plausible but incorrect)
Lang-35 (ARJA correct)
Lang-39 (ARJA plausible but incorrect)
Lang-41 (ARJA plausible but incorrect)
Lang-43 (ARJA correct)
Lang-45 (ARJA correct)
Lang-46 (ARJA plausible but incorrect)
Lang-50 (ARJA plausible but incorrect)
Lang-51 (ARJA plausible but incorrect)
Lang-55 (ARJA plausible but incorrect)
Lang-59 (ARJA plausible but incorrect)
Lang-61 (ARJA plausible but incorrect)

commons-math (18 subjects)

Math-2 (ARJA plausible but incorrect)
Math-8 (ARJA plausible but incorrect)
Math-20 (ARJA plausible but incorrect)
Math-22 (ARJA correct)
Math-39 (ARJA correct)
Math-49 (ARJA plausible but incorrect)
Math-50 (ARJA correct)
Math-53 (ARJA correct)
Math-56 (ARJA plausible but incorrect)
Math-58 (ARJA correct)
Math-60 (ARJA plausible but incorrect)
Math-70 (ARJA correct)
Math-73 (ARJA correct)
Math-74 (ARJA plausible but incorrect)
Math-81 (ARJA plausible but incorrect)

→ only one failing test case; a similar one has already been added to the test suite in Defects4J

Math-95 (ARJA plausible but incorrect)
Math-98 (ARJA correct)
Math-103 (ARJA plausible but incorrect)

joda-time (3 subjects)

Time-4 (ARJA plausible but incorrect)
Time-11 (ARJA plausible but incorrect)
Time-14 (ARJA plausible but incorrect)

jfreechart (3 subjects)

Chart-1 (ARJA plausible but incorrect)
Chart-5 (ARJA correct)
Chart-12 (ARJA correct)

Not Supported ARJA Plausible Subjects (16)

The following list includes subjects, for which the bug report does not contain sufficient information to formulate a meaningful assertion. Total count: 16 subjects.

Lang-53 (ARJA implausible)

→ only one failing test case, which is already added to the test suite in Defects4J

Lang-60 (ARJA plausible but incorrect)

While fixing LANG-294 I noticed that there are two other places in StrBuilder that reference thisBuf.length and unless I'm mistaken they shouldn't.

→ unclear how to formulate an assertion because the reporter just reports that some code needs change without any problem description

Lang-63 (ARJA plausible but incorrect)

→ only one failing test case, which is already added to the test suite in Defects4J

Math-5 (ARJA corret)

→ only one failing test case, which is already added to the test suite in Defects4J

Math-6 (ARJA plausible but incorrect)

"The method LevenbergMarquardtOptimizer.getIterations() does not report the correct number of iterations; It always returns 0. A quick look at the code shows that only SimplexOptimizer calls BaseOptimizer.incrementEvaluationsCount()

I've put a test case below. Notice how the evaluations count is correctly incremented, but the iterations count is not."

The bug report says that the method always returns zero but does not say when it is correct and when it is incorrect. It provides a test case; however, this one is already included in the defects4j test suite.

Math-28 (ARJA plausible but incorrect)

SimplexSolver throws UnboundedSolutionException when trying to solve minimization linear programming problem. The number of exception thrown depends on the number of variables.

The bug report says that the SimplexSolver throws an UnboundedSolutionException, but it remains unclear when it is expected and when unexpected. So we might not be able to write a general oracle. Also the test throws a different exception: MaxCountExceededException. Both extend the MathIllegalStateException, which is expected “if no solution fulfilling the constraints can be found in the allowed number of iterations”.

Math-31 (ARJA plausible but incorrect)

→ Not enough information for an assertion.

Math-68 (ARJA plausible but incorrect)

LevenbergMarquardtOptimizer ignores the VectorialConvergenceChecker parameter passed to it. This makes it hard to specify custom stopping criteria for the optimizer.

→ The bug report does not provide enough information for an assertion.

Math-71 (ARJA plausible but incorrect)

→ only one failing test case, which has already been added to the test suite in Defects4J

Math-80 (ARJA plausible but incorrect)

Some results computed by EigenDecompositionImpl are wrong. The following case computed by Fortran Lapack fails with version 2.0

→ only one failing test case, which is already added to the test suite in Defects4J

Math-82 (ARJA plausible but incorrect)

→ only one failing test case, which is already added to the test suite in Defects4J

Math-84 (ARJA plausible but incorrect)

MultiDirectional.iterateSimplex loops forever if the starting point is the correct solution.

However, we cannot check for this. The class allows to set a maximum number of iterations, as done in the provided test case:

multiDirectional.setMaxIterations(100);
multiDirectional.setMaxEvaluations(1000);

The provided test throws: org.apache.commons.math.optimization.OptimizationException: org.apache.commons.math.MaxIterationsExceededException: Maximal number of iterations (100) exceeded. However, but generally, this does not have to be a bug…

Math-85 (ARJA plausible but incorrect)

The provided test case throws an ConvergenceException, which is generally not a bug. Without more information, we cannot formulate a general oracle. Looking at the developer-provided patch it gets clear that a ConvergenceException is not acceptable if fa or fb are 0, but this information is not included in the report.

Math-86 (ARJA correct)

And it should throw an exception but it does not. I tested the matrix in R and R's cholesky decomposition method returns that the matrix is not symmetric positive definite.

Condition for a check is not known; in fact this check is wrong in the current implementation and needs repair.

Chart-3 (ARJA correct)
  • Bug Report: UNKNOWN
Chart-7 (ARJA plausible but incorrect)
  • Bug Report: UNKNOWN
Chart-13 (ARJA plausible but incorrect)
  • Bug Report: UNKNOWN
Chart-15 (ARJA plausible but incorrect)
  • Bug Report: UNKNOWN
Chart-19 (ARJA plausible but incorrect)
  • Bug Report: UNKNOWN
Chart-25 (ARJA plausible but incorrect)
  • Bug Report: UNKNOWN
Time-15 (ARJA correct)

→ only one failing test case, which is already added to the test suite in Defects4J

Other Supported Subjects (2)

Lang-19 (Example; ARJA implausible)

Additional subjects

Lang-6 (ARJA implausible, but in search space of ARJA-e)
Lang-57 (ARJA implausible, but in search space of ARJA-e)
Math-59 (ARJA implausible, but in search space of ARJA-e)

Additional non-supported subjects

Math-33 (ARJA implausible, but in search space of ARJA-e)

Methode SimplexSolver.optimeze(...) gives bad results with commons-math3-3.0 in a simple test problem. It works well in commons-math-2.2.

→ Bug report does not mention specifics. It only compares with a previous version that also does allow the setup of a simple check based on metamorphic relations because the previous version is not available within this codebase.

Math-80 (ARJA implausible, but in search space of ARJA-e)

→ the user reports wrong results for the EigenDecompositionImpl but there is no general condition to check. The report mentions the fortran library LAPACK version 3.2.1 which could generally serve as reference implementation, but this would go beyond a simple instrumentation.

Math-82 (ARJA implausible, but in search space of ARJA-e)

SimplexSolver didn't find the optimal solution.

→ Bug report explains a case where the solver does not find the optimal solution. However, there are no constraints or conditions that could be checked in our general oracle beyond the concrete test case.

Math-85 (ARJA implausible, but in search space of ARJA-e)

… gives the exception below. It should return (approx) 2.0000…

→ the bug report illustrates multiple examples, for which the NormalDistributionImpl.inverseCumulativeProbability(..) results in a MathException. However, the MathException is also sometimes expected, as the methods signature actually has the corresponding throw declaration. To define an oracle, we would need to know when it is expected or when it is not expected. But the report does not include this information.

Refactor Diffs

To allow Defects4j to work with Java 1.5+, we have refactored a few subjects, basically to replace the Enum identifiers with some non-keyword. The EvoRepair experiments were performed on the refactored subjects; so was every other tool that is compared with EvoRepair.

About

No description, website, or topics provided.

Resources

Readme
Activity
Custom properties

Stars

0 stars

Watchers

5 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

No packages published

Contributors 4

  •  
  •  
  •  
  •