Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consume a single conformance test ZIP #115

Merged
merged 27 commits into from
Dec 11, 2023
Merged
Show file tree
Hide file tree
Changes from 13 commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
0d909fb
Rename the conformance test framework subproject to conformance-tests…
netdpb Nov 17, 2023
117b862
Make framework and tests group `org.jspecify.conformance`.
netdpb Nov 20, 2023
0a829be
Add Maven publishing.
netdpb Nov 20, 2023
0291bbb
Set up the conformance test artifacts: A source jar of the assertions…
netdpb Nov 21, 2023
6bb674f
Remove unused plugin.
netdpb Nov 21, 2023
a636d06
Renaming and cleanup.
netdpb Nov 21, 2023
c16c57a
Merge branch 'main' of github.com:jspecify/nullness-checker-for-check…
netdpb Nov 29, 2023
31b9a26
Experimenting with using the distribution plugin to create a single z…
netdpb Dec 1, 2023
56cbce2
Merge branch 'main' of github.com:jspecify/nullness-checker-for-check…
netdpb Dec 4, 2023
8e40f4d
Depend on a ZIP artifact.
netdpb Dec 5, 2023
e140cc8
Fix up `UsesDep.java` and `Dep.java`. Update report.
netdpb Dec 5, 2023
1fd4d1a
Merge branch 'main' of github.com:jspecify/nullness-checker-for-check…
netdpb Dec 5, 2023
d3277d7
Clean up unused bits.
netdpb Dec 5, 2023
cbba1c3
Merge branch 'main' of github.com:jspecify/nullness-checker-for-check…
netdpb Dec 8, 2023
a7c8351
Run tests in details mode to try to figure out why they're failing on…
netdpb Dec 11, 2023
2d267a0
Examine test inputs to try to figure out why they're failing on GitHub.
netdpb Dec 11, 2023
ca706d7
Examine test inputs to try to figure out why they're failing on GitHub.
netdpb Dec 11, 2023
a210189
Examine test inputs to try to figure out why they're failing on GitHub.
netdpb Dec 11, 2023
f22e561
Examine test inputs to try to figure out why they're failing on GitHub.
netdpb Dec 11, 2023
f8384cf
Examine test inputs to try to figure out why they're failing on GitHub.
netdpb Dec 11, 2023
539b96a
Examine test inputs to try to figure out why they're failing on GitHub.
netdpb Dec 11, 2023
a82c954
Examine test inputs to try to figure out why they're failing on GitHub.
netdpb Dec 11, 2023
5737831
Found it! The system properties have to be set in the execution phase…
netdpb Dec 11, 2023
04e095d
Clean up the deps parsing.
netdpb Dec 11, 2023
8576b04
Clean up the deps parsing.
netdpb Dec 11, 2023
f7ca607
Use the task as input instead of the directory.
netdpb Dec 11, 2023
fbb6229
Remove debug printing.
netdpb Dec 11, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 18 additions & 29 deletions build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -15,12 +15,7 @@ repositories {

configurations {
errorproneJavac
conformanceTestAssertions {
attributes {
attribute(DocsType.DOCS_TYPE_ATTRIBUTE, objects.named(DocsType, DocsType.SOURCES))
}
}
conformanceTestDeps
conformanceTestSuite
}

ext {
Expand All @@ -46,19 +41,8 @@ dependencies {
testImplementation libs.jspecify.conformanceTestFramework
testRuntimeOnly libs.jsr305 // jsr305 annotations are in some of the samples

conformanceTestAssertions("org.jspecify.conformance:conformance-tests:0.0.0-SNAPSHOT") {
capabilities {
// Depend on the assertion sources
requireCapability "org.jspecify.conformance:conformance-tests-assertions"
}
}

conformanceTestDeps("org.jspecify.conformance:conformance-tests:0.0.0-SNAPSHOT") {
capabilities {
// Depend on the assertion dependencies
requireCapability "org.jspecify.conformance:conformance-tests-deps"
}
}
// TODO: Depend on a group:artifact:version rather than a file.
conformanceTestSuite files("${jspecify.projectDir}/conformance-tests/build/distributions/conformance-tests-0.0.0-SNAPSHOT.zip")

errorproneJavac libs.errorProne.javac
errorprone libs.errorProne.core
Expand Down Expand Up @@ -144,18 +128,23 @@ TaskProvider<Test> conformanceTest(String name, String testDirectory, String rep
}
}

tasks.register('unzipConformanceTestAssertions', Copy) {
dependsOn configurations.conformanceTestAssertions
from zipTree(configurations.conformanceTestAssertions.singleFile)
exclude 'META-INF/'
into "${buildDir}/conformanceTests"
tasks.register('unzipConformanceTestSuite', Copy) {
// TODO: Don't explicitly depend on an included build's task.
dependsOn jspecify.task(':conformance-tests:build')
dependsOn configurations.conformanceTestSuite
from zipTree(configurations.conformanceTestSuite.singleFile)
into layout.buildDirectory.dir("conformanceTests")
}

conformanceTest('conformanceTest', "${buildDir}/conformanceTests/org/jspecify/conformance/tests", 'tests/ConformanceTest-report.txt').configure {
dependsOn 'unzipConformanceTestAssertions', configurations.conformanceTestDeps
inputs.files(configurations.conformanceTestDeps)
systemProperty("JSpecifyConformanceTest.testDeps", configurations.conformanceTestDeps.join(":"))
}
conformanceTest('conformanceTest',
"${unzipConformanceTestSuite.destinationDir}/assertions/org/jspecify/conformance/tests",
'tests/ConformanceTest-report.txt')
.configure {
dependsOn unzipConformanceTestSuite
inputs.dir(unzipConformanceTestSuite.destinationDir)
systemProperty("JSpecifyConformanceTest.testDeps",
fileTree("${unzipConformanceTestSuite.destinationDir}/deps").join(":"))
}

conformanceTest('conformanceTestOnSamples', "${jspecify.projectDir}/samples", 'tests/ConformanceTestOnSamples-report.txt')

Expand Down
4 changes: 3 additions & 1 deletion tests/ConformanceTest-report.txt
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# 10 pass; 6 fail; 16 total; 62.5% score
# 12 pass; 6 fail; 18 total; 66.7% score
PASS: Basic.java:26 test:expression-type:Object?:nullable
PASS: Basic.java:26 test:sink-type:Object!:return
PASS: Basic.java:26 test:cannot-convert:Object? to Object!
Expand All @@ -15,3 +15,5 @@ FAIL: Irrelevant.java:43 test:irrelevant-annotation:Nullable
FAIL: Irrelevant.java:47 test:irrelevant-annotation:NullMarked
FAIL: Irrelevant.java:49 test:irrelevant-annotation:NullUnmarked
FAIL: Irrelevant.java: no unexpected facts
PASS: UsesDep.java:24 test:cannot-convert:null? to Dep*
PASS: UsesDep.java: no unexpected facts
Loading