Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmdLineTester_callsitedbgddrext_openj9 test failing with mixed references build #11469

Closed
sharon-wang opened this issue Dec 12, 2020 · 49 comments · Fixed by #11691 or eclipse-omr/omr#5758

Comments

@sharon-wang
Copy link
Contributor

sharon-wang commented Dec 12, 2020

Failure link: https://ci.eclipse.org/openj9/job/Test_openjdk11_j9_sanity.functional_s390x_linux_mixed_cm_Nightly/2/tapResults/

  • BUILD_LIST: functional
  • TARGET: sanity.functional
  • Failed Platforms:
    • ppc64_aix_mixed_cm --> SPEC renamed to ppc64_aix_mixed
    • s390x_linux_mixed_cm --> SPEC renamed to s390x_linux_mixed
  • Specific test suite failing: cmdLineTester_callsitedbgddrext_openj9_0
  • Specific tests failing: !printallcallsites, !findallcallsites

Version info (only failing with mixedrefs CMake build)

openjdk version "11.0.10-internal" 2021-01-19
OpenJDK Runtime Environment (build 11.0.10-internal+0-adhoc.****.BuildJDK11s390xlinuxmixedcmNightly)
Eclipse OpenJ9 VM (build master-f2e8dfa4346, JRE 11 Linux s390x-64-Bit Compressed References 20201210_3 (JIT enabled, AOT enabled)
OpenJ9   - f2e8dfa4346
OMR      - 5c72bedfef9
JCL      - 5e81062d6d3 based on jdk-11.0.10+5)

Discussion: #9284 and https://openj9.slack.com/archives/C8312LCV9/p1607532450200200
Possibly related: adoptium/run-aqa#50

Mixed Refs PRs:

Optional info

  • intermittent failure: no
  • regression or new test: new failure only with mixed refs builds

To generate a "bad core file":

  1. Build with configurations --with-mixedrefs --with-cmake on BE machine, such as s390x_linux or ppc64_aix
  2. Generate core file via java -Xdump:system:events=vmstop -version

Failure output

Testing: Run !printallcallsites
Test start time: 2020/12/10 20:49:41 Coordinated Universal Time
Running command: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_mixed_cm_Nightly_testList_1/openjdkbinary/j2sdk-image/bin/jdmpview -core j9core.dmp
Time spent starting: 2 milliseconds
Time spent executing: 3048 milliseconds
Test result: FAILED
Output from test:
 [OUT] DTFJView version 4.29.5, using DTFJ version 1.12.29003
 [OUT] Loading image from DTFJ...
 [OUT] 
 [OUT] Warning: dump file was produced by a different, possibly incompatible, VM.
 [OUT]   core file: (unknown)
 [OUT]   this VM:   Eclipse OpenJ9 VM
 [OUT] 
 [OUT] For a list of commands, type "help"; for how to use "help", type "help help"
 [OUT] Available contexts (* = currently selected context) : 
 [OUT] 
 [OUT] Source : file:///home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_mixed_cm_Nightly_testList_1/openjdk-tests/TKG/test_output_16076320077235/cmdLineTester_callsitedbgddrext_openj9_0/j9core.dmp
 [OUT] 	*0 : JRE 11 Linux s390x-64 (build 11.0.10-internal+0-adhoc.jenkins.BuildJDK11s390xlinuxmixedcmNightly)
 [OUT] 
 [OUT] > DDR is not enabled for this core file, '!' commands are disabled
 [OUT] > 
>> Success condition was not found: [Output match: jvminit.c]
>> Failure condition was not found: [Output match: DDRInteractiveCommandException]
>> Failure condition was not found: [Output match: no shared cache]
>> Failure condition was not found: [Output match: unable to read]
>> Failure condition was not found: [Output match: could not read]
>> Failure condition was not found: [Output match: dump event]

Testing: Run !findallcallsites
Test start time: 2020/12/10 20:49:44 Coordinated Universal Time
Running command: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_mixed_cm_Nightly_testList_1/openjdkbinary/j2sdk-image/bin/jdmpview -core j9core.dmp
Time spent starting: 0 milliseconds
Time spent executing: 1680 milliseconds
Test result: FAILED
Output from test:
 [OUT] DTFJView version 4.29.5, using DTFJ version 1.12.29003
 [OUT] Loading image from DTFJ...
 [OUT] 
 [OUT] Warning: dump file was produced by a different, possibly incompatible, VM.
 [OUT]   core file: (unknown)
 [OUT]   this VM:   Eclipse OpenJ9 VM
 [OUT] 
 [OUT] For a list of commands, type "help"; for how to use "help", type "help help"
 [OUT] Available contexts (* = currently selected context) : 
 [OUT] 
 [OUT] Source : file:///home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_mixed_cm_Nightly_testList_1/openjdk-tests/TKG/test_output_16076320077235/cmdLineTester_callsitedbgddrext_openj9_0/j9core.dmp
 [OUT] 	*0 : JRE 11 Linux s390x-64 (build 11.0.10-internal+0-adhoc.jenkins.BuildJDK11s390xlinuxmixedcmNightly)
 [OUT] 
 [OUT] > DDR is not enabled for this core file, '!' commands are disabled
 [OUT] > 
>> Success condition was not found: [Output match: jvminit.c]
>> Failure condition was not found: [Output match: DDRInteractiveCommandException]
>> Failure condition was not found: [Output match: no shared cache]
>> Failure condition was not found: [Output match: unable to read]
>> Failure condition was not found: [Output match: could not read]
>> Failure condition was not found: [Output match: dump event]


---TEST RESULTS---
Number of PASSED tests: 3 out of 5
Number of FAILED tests: 2 out of 5

---SUMMARY OF FAILED TESTS---
Run !printallcallsites
Run !findallcallsites

@sophia-guo
Copy link
Contributor

This is not limited to mixed references build. adoptium/run-aqa#50
adoptium/aqa-tests#2100

@pshipton
Copy link
Member

pshipton commented Dec 12, 2020

If you take the zlinux mixed build, create a core file, and open it with jdmpview using the -verbose option, it shows the reason for the failure, UnsupportedOperationException: GC policy not supported.
Also note only the BE platforms are failing.

java.lang.ExceptionInInitializerError
at java.base/java.lang.J9VMInternals.ensureError(J9VMInternals.java:185)
at java.base/java.lang.J9VMInternals.recordInitializationFailure(J9VMInternals.java:174)
at com.ibm.j9ddr.vm29.tools.ddrinteractive.GetCommandsTask.run(GetCommandsTask.java:184)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.ibm.j9ddr.vm29.j9.VMData.bootstrapClass(VMData.java:65)
at com.ibm.j9ddr.vm29.j9.VMData.bootstrapRelative(VMData.java:44)
at com.ibm.j9ddr.tools.ddrinteractive.Context.getContextCommands(Context.java:97)
at com.ibm.j9ddr.tools.ddrinteractive.Context.refreshCommandList(Context.java:87)
at com.ibm.j9ddr.tools.ddrinteractive.Context.(Context.java:68)
at com.ibm.j9ddr.tools.ddrinteractive.DDRInteractive.addDDRContextFromDTFJ(DDRInteractive.java:226)
at com.ibm.j9ddr.tools.ddrinteractive.DDRInteractive.(DDRInteractive.java:155)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
at openj9.dtfjview/com.ibm.jvm.dtfjview.CombinedContext.startDDRInteractiveSession(CombinedContext.java:219)
at openj9.dtfjview/com.ibm.jvm.dtfjview.commands.OpenCommand.createCombinedContext(OpenCommand.java:251)
at openj9.dtfjview/com.ibm.jvm.dtfjview.commands.OpenCommand.createContexts(OpenCommand.java:200)
at openj9.dtfjview/com.ibm.jvm.dtfjview.commands.OpenCommand.imagesFromCommandLine(OpenCommand.java:123)
at openj9.dtfjview/com.ibm.jvm.dtfjview.commands.OpenCommand.run(OpenCommand.java:83)
at openj9.dtfj/com.ibm.java.diagnostics.utils.Context.tryCommand(Context.java:141)
at openj9.dtfj/com.ibm.java.diagnostics.utils.Context.execute(Context.java:97)
at openj9.dtfjview/com.ibm.jvm.dtfjview.CombinedContext.execute(CombinedContext.java:173)
at openj9.dtfjview/com.ibm.jvm.dtfjview.CombinedContext.execute(CombinedContext.java:157)
at openj9.dtfjview/com.ibm.jvm.dtfjview.Session.imageFromCommandLine(Session.java:610)
at openj9.dtfjview/com.ibm.jvm.dtfjview.Session.sessionInit(Session.java:226)
at openj9.dtfjview/com.ibm.jvm.dtfjview.Session.(Session.java:175)
at openj9.dtfjview/com.ibm.jvm.dtfjview.Session.getInstance(Session.java:171)
at openj9.dtfjview/com.ibm.jvm.dtfjview.DTFJView.launch(DTFJView.java:51)
at openj9.dtfjview/com.ibm.jvm.dtfjview.DTFJView.main(DTFJView.java:46)
Caused by: java.lang.UnsupportedOperationException: GC policy not supported
at com.ibm.j9ddr.vm29.j9.gc.GCHeapMap.from(GCHeapMap.java:104)
at com.ibm.j9ddr.vm29.tools.ddrinteractive.commands.MarkMapCommand.(MarkMapCommand.java:51)
... 31 more

@pshipton
Copy link
Member

@sophia-guo you should open a new issue for the aqa failure, it's a different problem. This issue is specific to the mixed builds.

@sharon-wang
Copy link
Contributor Author

Still working on identifying the root problem here.

I added some prints to check the GCExtensions configurations in debugtools/DDR_VM/src/com/ibm/j9ddr/vm29/j9/gc/GCExtensions.java.

With a core file generated by a mixed refs build on a BE platform, the output is false for all configurations/settings:

J9BuildFlags.gc_combinationSpec: true
isSegregatedHeap: false
isMetronomeGC: false
isStandardGC: false
isVLHGC: false
scavengerEnabled: false

However, the expectation is to see:

J9BuildFlags.gc_combinationSpec: true
isSegregatedHeap: false
isMetronomeGC: false
isStandardGC: true // <--- set
isVLHGC: false
scavengerEnabled: true // <--- set

It seems that something might be going wrong in the initialization of GCExtensions or while the GCExtensions configurations are getting set. However, printing out the GCExtensions settings while the VM is running normally (not in jdmpview) yields the correct settings (isStandardGC: true, scavengerEnabled: true). It appears this problem is isolated to jdmpview.

I'm looking through the DDR/jdmpview code and trying to run some dump commands on the core file for more insights, but I'm not very familiar with jdmpview and welcome any additional pointers/suggestions!

@pshipton
Copy link
Member

Doesn't seem like it would be the same issue, but FYI #11639
@keithc-ca

How do you print the J9BuildFlags? Try dumping the raw memory instead of using DDR to print the values.

@keithc-ca
Copy link
Contributor

If you can find the blob in the core file, you could extract it with dd and then use blob_reader to check the flags.
If you have access to the SDK that produced the core file, you can run blob_reader on j9ddr.dat from that SDK.

@keithc-ca
Copy link
Contributor

There's also the !buildflags command in jdmpview.

@sharon-wang
Copy link
Contributor Author

#3837 seems relevant to this issue.

I was able to print the flags in j9ddr.dat from the SDK using omr_blob_reader. So far I don't see much cause for concern in the flags, versus the blob I generated with a passing "hacked" UMA mixed build.

I'm not able to use !buildflags with the problematic core: DDR is not enabled for this core file, '!' commands are disabled.

@tajila
Copy link
Contributor

tajila commented Jan 14, 2021

Buildflags output

> !buildflags
EsExtraVersionString_DEFINED = true
EsVersionMajor = true
EsVersionMajor_DEFINED = true
EsVersionMinor = true
EsVersionMinor_DEFINED = true
EsVersionString_DEFINED = true
J9VM_ARCH_AARCH64 = false
J9VM_ARCH_AARCH64_DEFINED = false
J9VM_ARCH_ARM = false
J9VM_ARCH_ARM_DEFINED = false
J9VM_ARCH_POWER = false
J9VM_ARCH_POWER_DEFINED = false
J9VM_ARCH_RISCV = false
J9VM_ARCH_RISCV_DEFINED = false
J9VM_ARCH_S390_DEFINED = true
J9VM_ARCH_X86 = false
J9VM_ARCH_X86_DEFINED = false
J9VM_BUILD_AUTOBUILD = false
J9VM_BUILD_AUTOBUILD_DEFINED = false
J9VM_BUILD_CMAKE = false
J9VM_BUILD_CMAKE_DEFINED = false
J9VM_BUILD_DROP_TO_HURSLEY = false
J9VM_BUILD_DROP_TO_HURSLEY_DEFINED = false
J9VM_BUILD_DROP_TO_PHOENIX = false
J9VM_BUILD_DROP_TO_PHOENIX_DEFINED = false
J9VM_BUILD_DROP_TO_TORONTO = false
J9VM_BUILD_DROP_TO_TORONTO_DEFINED = false
J9VM_BUILD_FIPS = false
J9VM_BUILD_FIPS_DEFINED = false
J9VM_BUILD_GC_CONTINUOUS = false
J9VM_BUILD_GC_CONTINUOUS_DEFINED = false
J9VM_BUILD_J2ME = false
J9VM_BUILD_J2ME_DEFINED = false
J9VM_BUILD_J2SE = false
J9VM_BUILD_J2SE_DEFINED = false
J9VM_BUILD_J9VM_DOC = false
J9VM_BUILD_J9VM_DOC_DEFINED = false
J9VM_BUILD_JAVA5 = false
J9VM_BUILD_JAVA5_DEFINED = false
J9VM_BUILD_JAVA6 = false
J9VM_BUILD_JAVA60_26 = false
J9VM_BUILD_JAVA60_26_DEFINED = false
J9VM_BUILD_JAVA6PROXY = false
J9VM_BUILD_JAVA6PROXY_DEFINED = false
J9VM_BUILD_JAVA6_DEFINED = false
J9VM_BUILD_JAVA7 = true
J9VM_BUILD_JAVA70_27 = false
J9VM_BUILD_JAVA70_27_DEFINED = false
J9VM_BUILD_JAVA7BASIC = false
J9VM_BUILD_JAVA7BASIC_DEFINED = false
J9VM_BUILD_JAVA7RAW = false
J9VM_BUILD_JAVA7RAW_DEFINED = false
J9VM_BUILD_JAVA7_DEFINED = true
J9VM_BUILD_JAVA8 = false
J9VM_BUILD_JAVA8RAW = false
J9VM_BUILD_JAVA8RAW_DEFINED = false
J9VM_BUILD_JAVA8_DEFINED = false
J9VM_BUILD_JAVA9 = false
J9VM_BUILD_JAVA9_DEFINED = false
J9VM_BUILD_NEW_COMPILER = false
J9VM_BUILD_NEW_COMPILER_DEFINED = false
J9VM_BUILD_OPENJ9 = false
J9VM_BUILD_OPENJ9_DEFINED = false
J9VM_BUILD_OUNCEMAKE = false
J9VM_BUILD_OUNCEMAKE_DEFINED = false
J9VM_BUILD_PRODUCT = false
J9VM_BUILD_PRODUCT_DEFINED = false
J9VM_BUILD_REALTIME = false
J9VM_BUILD_REALTIME_DEFINED = false
J9VM_BUILD_STAGE_OTTAWA_VMLAB = false
J9VM_BUILD_STAGE_OTTAWA_VMLAB_DEFINED = false
J9VM_BUILD_STAGE_TORONTO_LAB = false
J9VM_BUILD_STAGE_TORONTO_LAB_DEFINED = false
J9VM_BUILD_UMA = false
J9VM_BUILD_UMA_DEFINED = false
J9VM_BUILD_VM_CONTINUOUS = false
J9VM_BUILD_VM_CONTINUOUS_DEFINED = false
J9VM_BUILD__SE6_PACKAGE = false
J9VM_BUILD__SE6_PACKAGE_DEFINED = false
J9VM_BUILD__VS12_AND_HIGHER = false
J9VM_BUILD__VS12_AND_HIGHER_DEFINED = false
J9VM_COMPILER_PROMOTION = false
J9VM_COMPILER_PROMOTION_DEFINED = false
J9VM_DANGER_MEMLEAKS_BROKEN = false
J9VM_DANGER_MEMLEAKS_BROKEN_DEFINED = false
J9VM_ENV_ADVANCE_TOOLCHAIN = false
J9VM_ENV_ADVANCE_TOOLCHAIN_DEFINED = false
J9VM_ENV_CALL_VIA_TABLE = false
J9VM_ENV_CALL_VIA_TABLE_DEFINED = false
J9VM_ENV_DATA64_DEFINED = true
J9VM_ENV_DLPAR = false
J9VM_ENV_DLPAR_DEFINED = false
J9VM_ENV_GCC = false
J9VM_ENV_GCC_DEFINED = false
J9VM_ENV_HAS_FPU = true
J9VM_ENV_HAS_FPU_DEFINED = true
J9VM_ENV_LITTLE_ENDIAN_DEFINED = false
J9VM_ENV_SHARED_LIBS_CALLEE_GLOBAL_TABLE_SETUP = true
J9VM_ENV_SHARED_LIBS_CALLEE_GLOBAL_TABLE_SETUP_DEFINED = true
J9VM_ENV_SHARED_LIBS_USE_GLOBAL_TABLE = true
J9VM_ENV_SHARED_LIBS_USE_GLOBAL_TABLE_DEFINED = true
J9VM_ENV_SSE2_SUPPORT_DETECTION = false
J9VM_ENV_SSE2_SUPPORT_DETECTION_DEFINED = false
J9VM_ENV_Z_TPF = false
J9VM_ENV_Z_TPF_DEFINED = false
J9VM_GC_ADAPTIVE_TENURING = true
J9VM_GC_ADAPTIVE_TENURING_DEFINED = true
J9VM_GC_ALIGN_OBJECTS = false
J9VM_GC_ALIGN_OBJECTS_DEFINED = false
J9VM_GC_ALLOCATION_TAX = true
J9VM_GC_ALLOCATION_TAX_DEFINED = true
J9VM_GC_ALWAYS_CALL_OBJECT_ACCESS_BARRIER = false
J9VM_GC_ALWAYS_CALL_OBJECT_ACCESS_BARRIER_DEFINED = false
J9VM_GC_ALWAYS_CALL_WRITE_BARRIER = false
J9VM_GC_ALWAYS_CALL_WRITE_BARRIER_DEFINED = false
J9VM_GC_BATCH_CLEAR_TLH = true
J9VM_GC_BATCH_CLEAR_TLH_DEFINED = true
J9VM_GC_CLASSES_ON_HEAP = true
J9VM_GC_CLASSES_ON_HEAP_DEFINED = true
J9VM_GC_COMBINATION_SPEC_DEFINED = true
J9VM_GC_COMPRESSED_POINTERS_DEFINED = true
J9VM_GC_COMPRESSED_POINTER_BARRIER = false
J9VM_GC_COMPRESSED_POINTER_BARRIER_DEFINED = false
J9VM_GC_CONCURRENT_SWEEP = true
J9VM_GC_CONCURRENT_SWEEP_DEFINED = true
J9VM_GC_DEBUG_ASSERTS = true
J9VM_GC_DEBUG_ASSERTS_DEFINED = true
J9VM_GC_DYNAMIC_CLASS_UNLOADING_DEFINED = true
J9VM_GC_DYNAMIC_NEW_SPACE_SIZING = true
J9VM_GC_DYNAMIC_NEW_SPACE_SIZING_DEFINED = true
J9VM_GC_ENABLE_DOUBLE_MAP = true
J9VM_GC_ENABLE_DOUBLE_MAP_DEFINED = true
J9VM_GC_FINALIZATION_DEFINED = true
J9VM_GC_FRAGMENTED_HEAP = true
J9VM_GC_FRAGMENTED_HEAP_DEFINED = true
J9VM_GC_GENERATIONAL_DEFINED = true
J9VM_GC_HEAP_CARD_TABLE = true
J9VM_GC_HEAP_CARD_TABLE_DEFINED = true
J9VM_GC_INLINED_ALLOC_FIELDS_DEFINED = true
J9VM_GC_JNI_ARRAY_CACHE = true
J9VM_GC_JNI_ARRAY_CACHE_DEFINED = true
J9VM_GC_LARGE_OBJECT_AREA = true
J9VM_GC_LARGE_OBJECT_AREA_DEFINED = true
J9VM_GC_LEAF_BITS_DEFINED = true
J9VM_GC_MINIMUM_OBJECT_SIZE_DEFINED = true
J9VM_GC_MODRON_COMPACTION_DEFINED = true
J9VM_GC_MODRON_CONCURRENT_MARK = true
J9VM_GC_MODRON_CONCURRENT_MARK_DEFINED = true
J9VM_GC_MODRON_GC = true
J9VM_GC_MODRON_GC_DEFINED = true
J9VM_GC_MODRON_SCAVENGER_DEFINED = true
J9VM_GC_MODRON_STANDARD_DEFINED = true
J9VM_GC_MODRON_TRACE = true
J9VM_GC_MODRON_TRACE_DEFINED = true
J9VM_GC_MODRON_VERBOSE = true
J9VM_GC_MODRON_VERBOSE_DEFINED = true
J9VM_GC_NEW_SPINLOCK_SUPPORT = false
J9VM_GC_NEW_SPINLOCK_SUPPORT_DEFINED = false
J9VM_GC_NON_ZERO_TLH_DEFINED = true
J9VM_GC_OBJECT_ACCESS_BARRIER = true
J9VM_GC_OBJECT_ACCESS_BARRIER_DEFINED = true
J9VM_GC_REALTIME_DEFINED = true
J9VM_GC_SEGREGATED_HEAP_DEFINED = true
J9VM_GC_STRICT_OMR = false
J9VM_GC_STRICT_OMR_DEFINED = false
J9VM_GC_SUBPOOLS = false
J9VM_GC_SUBPOOLS_ALIAS = false
J9VM_GC_SUBPOOLS_ALIAS_DEFINED = false
J9VM_GC_SUBPOOLS_DEFINED = false
J9VM_GC_THREAD_LOCAL_HEAP = true
J9VM_GC_THREAD_LOCAL_HEAP_DEFINED = true
J9VM_GC_TILTED_NEW_SPACE = true
J9VM_GC_TILTED_NEW_SPACE_DEFINED = true
J9VM_GC_TLH_PREFETCH_FTA = true
J9VM_GC_TLH_PREFETCH_FTA_DEFINED = true
J9VM_GC_USE_INLINE_ALLOCATE = true
J9VM_GC_USE_INLINE_ALLOCATE_DEFINED = true
J9VM_GC_VERIFY_ACCESS_BARRIER = false
J9VM_GC_VERIFY_ACCESS_BARRIER_DEFINED = false
J9VM_GC_VLHGC_DEFINED = true
J9VM_INTERP_AOT_COMPILE_SUPPORT = true
J9VM_INTERP_AOT_COMPILE_SUPPORT_DEFINED = true
J9VM_INTERP_AOT_RUNTIME_SUPPORT = true
J9VM_INTERP_AOT_RUNTIME_SUPPORT_DEFINED = true
J9VM_INTERP_ATOMIC_FREE_JNI = false
J9VM_INTERP_ATOMIC_FREE_JNI_DEFINED = false
J9VM_INTERP_ATOMIC_FREE_JNI_USES_FLUSH = false
J9VM_INTERP_ATOMIC_FREE_JNI_USES_FLUSH_DEFINED = false
J9VM_INTERP_BYTECODE_PREVERIFICATION = true
J9VM_INTERP_BYTECODE_PREVERIFICATION_DEFINED = true
J9VM_INTERP_BYTECODE_VERIFICATION = true
J9VM_INTERP_BYTECODE_VERIFICATION_DEFINED = true
J9VM_INTERP_COMPRESSED_OBJECT_HEADER_DEFINED = true
J9VM_INTERP_CUSTOM_SPIN_OPTIONS = true
J9VM_INTERP_CUSTOM_SPIN_OPTIONS_DEFINED = true
J9VM_INTERP_DEBUG_SUPPORT_DEFINED = true
J9VM_INTERP_ENABLE_JIT_ON_DESKTOP = true
J9VM_INTERP_ENABLE_JIT_ON_DESKTOP_DEFINED = true
J9VM_INTERP_FLAGS_IN_CLASS_SLOT_DEFINED = true
J9VM_INTERP_FLOATMATHLIB_TRACING = false
J9VM_INTERP_FLOATMATHLIB_TRACING_DEFINED = false
J9VM_INTERP_FLOATMATH_TRACING = false
J9VM_INTERP_FLOATMATH_TRACING_DEFINED = false
J9VM_INTERP_FLOAT_SUPPORT = true
J9VM_INTERP_FLOAT_SUPPORT_DEFINED = true
J9VM_INTERP_GP_HANDLER = true
J9VM_INTERP_GP_HANDLER_DEFINED = true
J9VM_INTERP_GROWABLE_STACKS_DEFINED = true
J9VM_INTERP_HOT_CODE_REPLACEMENT_DEFINED = true
J9VM_INTERP_JIT_ON_BY_DEFAULT = true
J9VM_INTERP_JIT_ON_BY_DEFAULT_DEFINED = true
J9VM_INTERP_JNI_SUPPORT = true
J9VM_INTERP_JNI_SUPPORT_DEFINED = true
J9VM_INTERP_MINIMAL_JCL = false
J9VM_INTERP_MINIMAL_JCL_DEFINED = false
J9VM_INTERP_MINIMAL_JNI = false
J9VM_INTERP_MINIMAL_JNI_DEFINED = false
J9VM_INTERP_NATIVE_SUPPORT_DEFINED = true
J9VM_INTERP_NEW_HEADER_SHAPE = true
J9VM_INTERP_NEW_HEADER_SHAPE_DEFINED = true
J9VM_INTERP_PROFILING_BYTECODES = true
J9VM_INTERP_PROFILING_BYTECODES_DEFINED = true
J9VM_INTERP_ROMABLE_AOT_SUPPORT = false
J9VM_INTERP_ROMABLE_AOT_SUPPORT_DEFINED = false
J9VM_INTERP_SIG_QUIT_THREAD = true
J9VM_INTERP_SIG_QUIT_THREAD_DEFINED = true
J9VM_INTERP_SIG_QUIT_THREAD_USES_SEMAPHORES = true
J9VM_INTERP_SIG_QUIT_THREAD_USES_SEMAPHORES_DEFINED = true
J9VM_INTERP_SMALL_MONITOR_SLOT_DEFINED = true
J9VM_INTERP_TRACING = false
J9VM_INTERP_TRACING_DEFINED = false
J9VM_INTERP_TWO_PASS_EXCLUSIVE = false
J9VM_INTERP_TWO_PASS_EXCLUSIVE_DEFINED = false
J9VM_INTERP_UPDATE_VMCTRACING = false
J9VM_INTERP_UPDATE_VMCTRACING_DEFINED = false
J9VM_INTERP_USE_SPLIT_SIDE_TABLES = true
J9VM_INTERP_USE_SPLIT_SIDE_TABLES_DEFINED = true
J9VM_INTERP_USE_UNSAFE_HELPER = true
J9VM_INTERP_USE_UNSAFE_HELPER_DEFINED = true
J9VM_INTERP_VERBOSE = true
J9VM_INTERP_VERBOSE_DEFINED = true
J9VM_IVE_JXE_FILE_RELOCATOR = true
J9VM_IVE_JXE_FILE_RELOCATOR_DEFINED = true
J9VM_IVE_JXE_IN_PLACE_RELOCATOR = true
J9VM_IVE_JXE_IN_PLACE_RELOCATOR_DEFINED = true
J9VM_IVE_JXE_NATIVES = true
J9VM_IVE_JXE_NATIVES_DEFINED = true
J9VM_IVE_JXE_OERELOCATOR = true
J9VM_IVE_JXE_OERELOCATOR_DEFINED = true
J9VM_IVE_JXE_STREAMING_RELOCATOR = true
J9VM_IVE_JXE_STREAMING_RELOCATOR_DEFINED = true
J9VM_IVE_MEMORY_SPACE_HELPERS = false
J9VM_IVE_MEMORY_SPACE_HELPERS_DEFINED = false
J9VM_IVE_ROM_IMAGE_HELPERS = true
J9VM_IVE_ROM_IMAGE_HELPERS_DEFINED = true
J9VM_JIT_32BIT_USES64BIT_REGISTERS_DEFINED = false
J9VM_JIT_CLASS_UNLOAD_RWMONITOR = true
J9VM_JIT_CLASS_UNLOAD_RWMONITOR_DEFINED = true
J9VM_JIT_C_HELPERS = true
J9VM_JIT_C_HELPERS_DEFINED = true
J9VM_JIT_DYNAMIC_LOOP_TRANSFER = true
J9VM_JIT_DYNAMIC_LOOP_TRANSFER_DEFINED = true
J9VM_JIT_FREE_SYSTEM_STACK_POINTER = false
J9VM_JIT_FREE_SYSTEM_STACK_POINTER_DEFINED = false
J9VM_JIT_FULL_SPEED_DEBUG_DEFINED = true
J9VM_JIT_GC_ON_RESOLVE_SUPPORT = true
J9VM_JIT_GC_ON_RESOLVE_SUPPORT_DEFINED = true
J9VM_JIT_IA32_FIXED_FRAME = false
J9VM_JIT_IA32_FIXED_FRAME_DEFINED = false
J9VM_JIT_MICRO_JIT = false
J9VM_JIT_MICRO_JIT_DEFINED = false
J9VM_JIT_NATHELP_USES_CLASS_OBJECTS = false
J9VM_JIT_NATHELP_USES_CLASS_OBJECTS_DEFINED = false
J9VM_JIT_NEEDS_TRAMPOLINES = false
J9VM_JIT_NEEDS_TRAMPOLINES_DEFINED = false
J9VM_JIT_NEW_DUAL_HELPERS = true
J9VM_JIT_NEW_DUAL_HELPERS_DEFINED = true
J9VM_JIT_NEW_INSTANCE_PROTOTYPE = true
J9VM_JIT_NEW_INSTANCE_PROTOTYPE_DEFINED = true
J9VM_JIT_ON_STACK_REPLACEMENT = false
J9VM_JIT_ON_STACK_REPLACEMENT_DEFINED = false
J9VM_JIT_REQUIRES_TRAP_HANDLER = false
J9VM_JIT_REQUIRES_TRAP_HANDLER_DEFINED = false
J9VM_JIT_RUNTIME_INSTRUMENTATION = true
J9VM_JIT_RUNTIME_INSTRUMENTATION_DEFINED = true
J9VM_JIT_SMALL = false
J9VM_JIT_SMALL_DEFINED = false
J9VM_JIT_SUPPORTS_DIRECT_JNI = true
J9VM_JIT_SUPPORTS_DIRECT_JNI_DEFINED = true
J9VM_JIT_TRANSACTION_DIAGNOSTIC_THREAD_BLOCK = true
J9VM_JIT_TRANSACTION_DIAGNOSTIC_THREAD_BLOCK_DEFINED = true
J9VM_MATH_DIRECT_HELPERS = false
J9VM_MATH_DIRECT_HELPERS_DEFINED = false
J9VM_OPT_ANNOTATIONS = true
J9VM_OPT_ANNOTATIONS_DEFINED = true
J9VM_OPT_BIG_INTEGER = true
J9VM_OPT_BIG_INTEGER_DEFINED = true
J9VM_OPT_CUDA = false
J9VM_OPT_CUDA_DEFINED = false
J9VM_OPT_DEBUG_INFO_SERVER_DEFINED = true
J9VM_OPT_DEBUG_JSR45_SUPPORT = true
J9VM_OPT_DEBUG_JSR45_SUPPORT_DEFINED = true
J9VM_OPT_DEPRECATED_METHODS = true
J9VM_OPT_DEPRECATED_METHODS_DEFINED = true
J9VM_OPT_DMA_NATIVES = false
J9VM_OPT_DMA_NATIVES_DEFINED = false
J9VM_OPT_DYNAMIC_LOAD_SUPPORT = true
J9VM_OPT_DYNAMIC_LOAD_SUPPORT_DEFINED = true
J9VM_OPT_FIPS = false
J9VM_OPT_FIPS_DEFINED = false
J9VM_OPT_FRAGMENT_RAM_CLASSES = true
J9VM_OPT_FRAGMENT_RAM_CLASSES_DEFINED = true
J9VM_OPT_INLINE_JSRS = true
J9VM_OPT_INLINE_JSRS_DEFINED = true
J9VM_OPT_INVARIANT_INTERNING = true
J9VM_OPT_INVARIANT_INTERNING_DEFINED = true
J9VM_OPT_JAVA_OFFLOAD_SUPPORT = false
J9VM_OPT_JAVA_OFFLOAD_SUPPORT_DEFINED = false
J9VM_OPT_JITSERVER = false
J9VM_OPT_JITSERVER_DEFINED = false
J9VM_OPT_JVMTI_DEFINED = true
J9VM_OPT_JXE_LOAD_SUPPORT = true
J9VM_OPT_JXE_LOAD_SUPPORT_DEFINED = true
J9VM_OPT_MEMORY_CHECK_SUPPORT = true
J9VM_OPT_MEMORY_CHECK_SUPPORT_DEFINED = true
J9VM_OPT_METHOD_HANDLE = true
J9VM_OPT_METHOD_HANDLE_COMMON = true
J9VM_OPT_METHOD_HANDLE_COMMON_DEFINED = true
J9VM_OPT_METHOD_HANDLE_DEFINED = true
J9VM_OPT_MODULE = true
J9VM_OPT_MODULE_DEFINED = true
J9VM_OPT_MULTI_LAYER_SHARED_CLASS_CACHE = true
J9VM_OPT_MULTI_LAYER_SHARED_CLASS_CACHE_DEFINED = true
J9VM_OPT_MULTI_VM = true
J9VM_OPT_MULTI_VM_DEFINED = true
J9VM_OPT_NATIVE_CHARACTER_CONVERTER = false
J9VM_OPT_NATIVE_CHARACTER_CONVERTER_DEFINED = false
J9VM_OPT_NATIVE_LOCALE_SUPPORT = false
J9VM_OPT_NATIVE_LOCALE_SUPPORT_DEFINED = false
J9VM_OPT_NEW_OBJECT_HASH = true
J9VM_OPT_NEW_OBJECT_HASH_DEFINED = true
J9VM_OPT_NEW_ROM_CLASS_BUILDER = true
J9VM_OPT_NEW_ROM_CLASS_BUILDER_DEFINED = true
J9VM_OPT_NO_CLASSLOADERS = false
J9VM_OPT_NO_CLASSLOADERS_DEFINED = false
J9VM_OPT_NRR = false
J9VM_OPT_NRR_DEFINED = false
J9VM_OPT_OPENJDK_METHODHANDLE = false
J9VM_OPT_OPENJDK_METHODHANDLE_DEFINED = false
J9VM_OPT_PACKED = false
J9VM_OPT_PACKED_DEFINED = false
J9VM_OPT_PANAMA = false
J9VM_OPT_PANAMA_DEFINED = false
J9VM_OPT_PHP_SUPPORT = false
J9VM_OPT_PHP_SUPPORT_DEFINED = false
J9VM_OPT_REFLECT = true
J9VM_OPT_REFLECT_DEFINED = true
J9VM_OPT_RESOURCE_MANAGED = false
J9VM_OPT_RESOURCE_MANAGED_DEFINED = false
J9VM_OPT_ROM_IMAGE_SUPPORT = true
J9VM_OPT_ROM_IMAGE_SUPPORT_DEFINED = true
J9VM_OPT_SHARED_CLASSES_DEFINED = true
J9VM_OPT_SIDECAR = true
J9VM_OPT_SIDECAR_DEFINED = true
J9VM_OPT_SRP_AVL_TREE_SUPPORT = true
J9VM_OPT_SRP_AVL_TREE_SUPPORT_DEFINED = true
J9VM_OPT_STRING_COMPRESSION = true
J9VM_OPT_STRING_COMPRESSION_DEFINED = true
J9VM_OPT_SWITCH_STACKS_FOR_SIGNAL_HANDLER = false
J9VM_OPT_SWITCH_STACKS_FOR_SIGNAL_HANDLER_DEFINED = false
J9VM_OPT_TEMP_NEW_INTERFACE_INVOCATION = false
J9VM_OPT_TEMP_NEW_INTERFACE_INVOCATION_DEFINED = false
J9VM_OPT_USE_FFI = true
J9VM_OPT_USE_FFI_DEFINED = true
J9VM_OPT_USE_FFI_ONLY = true
J9VM_OPT_USE_FFI_ONLY_DEFINED = true
J9VM_OPT_USE_OMR_DDR = true
J9VM_OPT_USE_OMR_DDR_DEFINED = true
J9VM_OPT_VALHALLA_VALUE_TYPES = false
J9VM_OPT_VALHALLA_VALUE_TYPES_DEFINED = false
J9VM_OPT_VEE_SUPPORT = false
J9VM_OPT_VEE_SUPPORT_DEFINED = false
J9VM_OPT_VM_LOCAL_STORAGE = true
J9VM_OPT_VM_LOCAL_STORAGE_DEFINED = true
J9VM_OPT_ZERO = false
J9VM_OPT_ZERO_DEFINED = false
J9VM_OPT_ZIP_SUPPORT = true
J9VM_OPT_ZIP_SUPPORT_DEFINED = true
J9VM_OPT_ZLIB_COMPRESSION = true
J9VM_OPT_ZLIB_COMPRESSION_DEFINED = true
J9VM_OPT_ZLIB_SUPPORT = true
J9VM_OPT_ZLIB_SUPPORT_DEFINED = true
J9VM_PORT_OMRSIG_SUPPORT = true
J9VM_PORT_OMRSIG_SUPPORT_DEFINED = true
J9VM_PORT_RUNTIME_INSTRUMENTATION = true
J9VM_PORT_RUNTIME_INSTRUMENTATION_DEFINED = true
J9VM_PORT_SIGNAL_SUPPORT = true
J9VM_PORT_SIGNAL_SUPPORT_DEFINED = true
J9VM_PORT_ZOS_CEEHDLRSUPPORT = false
J9VM_PORT_ZOS_CEEHDLRSUPPORT_DEFINED = false
J9VM_PROF_COUNT_ARGS_TEMPS = false
J9VM_PROF_COUNT_ARGS_TEMPS_DEFINED = false
J9VM_PROF_EVENT_REPORTING = true
J9VM_PROF_EVENT_REPORTING_DEFINED = true
J9VM_PROF_JVMTI = false
J9VM_PROF_JVMTI_DEFINED = false
J9VM_RAS_DUMP_AGENTS = true
J9VM_RAS_DUMP_AGENTS_DEFINED = true
J9VM_RAS_EYECATCHERS = true
J9VM_RAS_EYECATCHERS_DEFINED = true
J9VM_RAS_FATAL_ASSERT = false
J9VM_RAS_FATAL_ASSERT_DEFINED = false
J9VM_SIZE_OPTIMIZE_SEND_TARGETS = true
J9VM_SIZE_OPTIMIZE_SEND_TARGETS_DEFINED = true
J9VM_SIZE_SMALL_CODE = false
J9VM_SIZE_SMALL_CODE_DEFINED = false
J9VM_SIZE_SMALL_OS_STACK = false
J9VM_SIZE_SMALL_OS_STACK_DEFINED = false
J9VM_SIZE_SMALL_RAM = false
J9VM_SIZE_SMALL_RAM_DEFINED = false
J9VM_TEMP_ALIGN_CLASS_SLOT = false
J9VM_TEMP_ALIGN_CLASS_SLOT_DEFINED = false
J9VM_TEMP_KEEP_FLAGS_SLOT = false
J9VM_TEMP_KEEP_FLAGS_SLOT_DEFINED = false
J9VM_THR_ASYNC_NAME_UPDATE = true
J9VM_THR_ASYNC_NAME_UPDATE_DEFINED = true
J9VM_THR_EXTRA_CHECKS = false
J9VM_THR_EXTRA_CHECKS_DEFINED = false
J9VM_THR_JLM_HST = false
J9VM_THR_JLM_HST_DEFINED = false
J9VM_THR_LOCK_NURSERY_DEFINED = true
J9VM_THR_LOCK_NURSERY_FAT_ARRAYS = false
J9VM_THR_LOCK_NURSERY_FAT_ARRAYS_DEFINED = false
J9VM_THR_LOCK_RESERVATION_DEFINED = true
J9VM_THR_PREEMPTIVE = true
J9VM_THR_PREEMPTIVE_DEFINED = true
J9VM_THR_SMART_DEFLATION = true
J9VM_THR_SMART_DEFLATION_DEFINED = true
J9_COPYRIGHT_STRING_DEFINED = true
JAVA_SPEC_VERSION = true
JAVA_SPEC_VERSION_DEFINED = true
JAVA_SPEC_VERSION_STRING_DEFINED = true
arch_aarch64 = false
arch_arm = false
arch_power = false
arch_riscv = false
arch_s390 = true
arch_x86 = false
env_data64 = true
env_littleEndian = false
gc_combinationSpec = true
gc_compressedPointers = true
gc_dynamicClassUnloading = true
gc_finalization = true
gc_generational = true
gc_inlinedAllocFields = true
gc_leafBits = true
gc_minimumObjectSize = true
gc_modronCompaction = true
gc_modronScavenger = true
gc_modronStandard = true
gc_nonZeroTLH = true
gc_realtime = true
gc_segregatedHeap = true
gc_vlhgc = true
interp_compressedObjectHeader = true
interp_debugSupport = true
interp_flagsInClassSlot = true
interp_growableStacks = true
interp_hotCodeReplacement = true
interp_nativeSupport = true
interp_smallMonitorSlot = true
jit_32bitUses64bitRegisters = false
jit_fullSpeedDebug = true
opt_debugInfoServer = true
opt_jvmti = true
opt_sharedClasses = true
thr_lockNursery = true
thr_lockReservation = true

@tajila
Copy link
Contributor

tajila commented Jan 14, 2021

The important ones in this case:

J9VM_GC_SEGREGATED_HEAP_DEFINED = true
J9VM_GC_MODRON_STANDARD_DEFINED = true

@tajila
Copy link
Contributor

tajila commented Jan 14, 2021

I think the real problem is here:

> !MM_GCExtensions 0x3FF9C02A5E0
MM_GCExtensions at 0x3ff9c02a5e0 {
  Fields for MM_Base:
  Fields for MM_BaseVirtual:
	0x8: const U8* _typeId = !j9x 0x000003FFA20BA49E // "MM_GCExtensions"
  Fields for MM_GCExtensionsBase:
	0x10: bool _compressObjectReferences = false
	0x18: void* _guaranteedNurseryStart = !j9x 0xFFFFFFFFFFFFFFFF
        ...
	0x5860: struct J9Pool* _lightweightNonReentrantLockPool = !j9pool 0x0000000100000000
	0x5868: struct J9ThreadMonitor* _lightweightNonReentrantLockPoolMutex = !j9threadmonitor 0x0000000000000300
	0x5870: bool _isSegregatedHeap = false
	0x5871: bool _isVLHGC = false
	0x5872: bool _isMetronomeGC = false
	0x5873: bool _isStandardGC = false
	0x5878: U64 tlhMinimumSize = 0x0000000000001000 (4096)
	...
	0x6748: class MM_IdleGCManager* idleGCManager = !mm_idlegcmanager 0x0000000000000000
	0x6750: double maxRAMPercent = -7.627449093663844E-42
	0x6758: double initialRAMPercent = 1.3067E-319
}

We look at the runtime state of MM_GCExtensions to determine which markmap to use if gc_combinationSpec == true (which it is in this case). Since the runtime MM_GCExtensions has those fields set to false this causes the failure to launch DDR.

So the problem is not with DDR, we need to determine if the GC configuration code in OMR is being run properly.

MM_ConfigurationSegregated::initialize
MM_ConfigurationStandard::initialize

@keithc-ca
Copy link
Contributor

I ran the VM under gdb:

  • watched it set _isStandardGC = true
  • set a hardware watch to see if anyone modified it later
  • verified it was still set after -version output was generated

@tajila
Copy link
Contributor

tajila commented Jan 14, 2021

Okay, I think DDR is wrong now:

Here is the DDR output:

	0x5848: U64 freeMemoryProfileMaxSizeClasses = 0x000003FF9C007F08 (4396368822024)
	0x5850: volatile struct OMR_VMThread* gcExclusiveAccessThreadId = !omr_vmthread 0x000003FF9C034220
	0x5858: struct J9ThreadMonitor* gcExclusiveAccessMutex = !j9threadmonitor 0x000003FF9C007FB8
	0x5860: struct J9Pool* _lightweightNonReentrantLockPool = !j9pool 0x0000000100000000
	0x5868: struct J9ThreadMonitor* _lightweightNonReentrantLockPoolMutex = !j9threadmonitor 0x0000000000000300
	0x5870: bool _isSegregatedHeap = false
	0x5871: bool _isVLHGC = false
	0x5872: bool _isMetronomeGC = false
	0x5873: bool _isStandardGC = false
	0x5878: U64 tlhMinimumSize = 0x0000000000001000 (4096)

Here are the raw bytes annotated:

0x3FF9C02FE28 :  000003ff9c007f08 //freeMemoryProfileMaxSizeClasses
                 000003ff9c034220 //gcExclusiveAccessThreadId
0x3FF9C02FE38 :  000003ff9c007fb8 //gcExclusiveAccessMutex
                 0000000100000000 //_lightweightNonReentrantLockPool
0x3FF9C02FE48 :  0000000000000300 //_lightweightNonReentrantLockPoolMutex
                 0000000000020000 // [_isSegregatedHeap,_isVLHGC,_isMetronomeGC,_isStandardGC]
0x3FF9C02FE58 :  0000000000001000 //tlhMinimumSize

@tajila
Copy link
Contributor

tajila commented Jan 14, 2021

With Sharon's help we discovered that the offset shown in gdb is:

_isSegregatedHeap // offset 0x5860
_isVLHGC // offset 0x5861
_isVLHGC // offset 0x5862
_isStandardGC //offset 0x5863

This differs from the offsets DDR is reporting

@tajila
Copy link
Contributor

tajila commented Jan 15, 2021

Sharon and I have discovered that DDR has an extra field. _compressObjectReferences which shifts down the offsets of all the other fields in MM_GCExtensions causing the issue.

@tajila
Copy link
Contributor

tajila commented Jan 15, 2021

#if defined(OMR_GC_COMPRESSED_POINTERS) && defined(OMR_GC_FULL_POINTERS) && !defined(OMR_OVERRIDE_COMPRESS_OBJECT_REFERENCES)
	bool _compressObjectReferences;
#endif /* defined(OMR_GC_COMPRESSED_POINTERS) && defined(OMR_GC_FULL_POINTERS) && !defined(OMR_OVERRIDE_COMPRESS_OBJECT_REFERENCES) */

OMR_OVERRIDE_COMPRESS_OBJECT_REFERENCES should be defined in mixed builds, but DDR blob does not have it defined

@tajila
Copy link
Contributor

tajila commented Jan 15, 2021

blob file:

Struct name: MM_GCExtensionsBase
 superName: MM_BaseVirtual
 sizeOf: 25184
 fieldCount: 426
 constCount: 3
 Field declaredName: _compressObjectReferences
  declaredType: bool
  offset: 16

@keithc-ca
Copy link
Contributor

I suspect this is a consequence of not compiling all source files with the same macro definitions. If ddrgen first sees a DWARF description of MM_GCExtensionsBase which includes _compressObjectReferences, it will remember the offsets from that description and ignore later descriptions.

@sharon-wang
Copy link
Contributor Author

When is the DWARF description of MM_GCExtensionsBase available/used?

OMR_OVERRIDE_COMPRESS_OBJECT_REFERENCES is defined in the CMakeLists.txt for omrgc/omrgc_full libraries specifically. It is not defined at the same time as the other flags, such as OMR_GC_COMPRESSED_POINTERS, OMR_GC_FULL_POINTERS. Is it possible that OMR_OVERRIDE_COMPRESS_OBJECT_REFERENCES is not defined at the time the DWARF description is being read, i.e. the description is being read before omrgc/omrgc_full have been compiled?

@keithc-ca
Copy link
Contributor

ddrgen examines libj9gc.so (and other libraries). If any object file linked into that shared library was compiled such that

  defined(OMR_GC_COMPRESSED_POINTERS) && defined(OMR_GC_FULL_POINTERS) && !defined(OMR_OVERRIDE_COMPRESS_OBJECT_REFERENCES)

is true, then the unexpected field will be found and have consequences for the offsets of following fields.

@sharon-wang
Copy link
Contributor Author

These changes to add a dummy function work to fix this issue: master...sharon-wang:crdummy and eclipse-omr/omr@master...sharon-wang:crdummy.

The reason why && !defined(OMR_OVERRIDE_COMPRESS_OBJECT_REFERENCES) was originally added to

#if defined(OMR_GC_COMPRESSED_POINTERS) && defined(OMR_GC_FULL_POINTERS) && !defined(OMR_OVERRIDE_COMPRESS_OBJECT_REFERENCES)
	bool _compressObjectReferences;
#endif /* defined(OMR_GC_COMPRESSED_POINTERS) && defined(OMR_GC_FULL_POINTERS) && !defined(OMR_OVERRIDE_COMPRESS_OBJECT_REFERENCES) */

is because the compiler complained about an unused field _compressObjectReferences, since it doesn't get used when OMR_OVERRIDE_COMPRESS_OBJECT_REFERENCES gets defined.

However, adding && !defined(OMR_OVERRIDE_COMPRESS_OBJECT_REFERENCES) breaks DDR (for some reason on BE platforms only) because it doesn't see the flag as defined when it is reading header files. The dummy function makes it so we don't have to include && !defined(OMR_OVERRIDE_COMPRESS_OBJECT_REFERENCES) to guard the field definition and works around the unused field issue.

Does this seem like a reasonable fix? It is a bit of a hack...

FYI @gacholio.

@keithc-ca
Copy link
Contributor

If we decide that the 'dummy' functions are a reasonable solution, they shouldn't need to be defined (anywhere). I don't see how a compiler could reasonably complain that a field is unused if there are any non-inline functions (or friends) declared in a class (it might be used by the implementation of that non-inline member (or accessed by a friend class)).

@sharon-wang
Copy link
Contributor Author

Maybe it's just the compiler on my mac, where I get

openj9-openjdk-jdk11/omr/gc/base/SlotObject.hpp:42:7: error: private field '_compressObjectReferences' is not used [-Werror,-Wunused-private-field]
        bool _compressObjectReferences;

if I just remove && !defined(OMR_OVERRIDE_COMPRESS_OBJECT_REFERENCES) (without any of the 'dummy' function changes).

However, it seems to work on other machines without the unused private field error. I can try testing on a few different platforms to see if this is specific to mac.

@sharon-wang
Copy link
Contributor Author

Removing && !defined(OMR_OVERRIDE_COMPRESS_OBJECT_REFERENCES) builds successfully on ppc aix, x86 linux, s390x linux, but results in the "private field is not used" error on mac -- so looks like this is specific to mac (tested with Apple clang version 12.0.0 (clang-1200.0.32.28) and Apple LLVM version 10.0.1 (clang-1001.0.46.4)).

The 'dummy' function could be ifdeffed on OSX, unless we want to explore a different fix.

@gacholio
Copy link
Contributor

If it's only the one compiler, a better solution might be to disable that warning.

@keithc-ca
Copy link
Contributor

The issue that will remain is inconsistent type definitions that will break DDR; see #11469 (comment).

@gacholio
Copy link
Contributor

If the override ifdef is removed, doesn't that take care of the inconsistency? DDR and the real compile will be the same.

@sharon-wang
Copy link
Contributor Author

Yes, if we remove && !defined(OMR_OVERRIDE_COMPRESS_OBJECT_REFERENCES), all types that currently conditionally define _compressObjectReferences will define it regardless of if we're building with mixedrefs or not. We will just need to disable the compiler error for OSX so that it can build successfully.

@keithc-ca
Copy link
Contributor

I added an extra field (for easy identification) next to _compressObjectReferences and then search through the resulting debug information with objdump -g. I only found one offender: runtime/ddr/gcddr.cpp.

@gacholio
Copy link
Contributor

So, compiling that file with the override would also fix this? That would be a better overall solution.

@keithc-ca
Copy link
Contributor

Yes, I think that would fix it.

I'd like to see a change that would catch any other mis-matches: Ideas on how to do that, anyone?

@keithc-ca
Copy link
Contributor

A second file that is missing that macro definition is omr/startup/omrvmstartup.cpp. That is built into a static library used many places. I'm not sure much will be left if we keep splitting libraries.

@sharon-wang
Copy link
Contributor Author

So we'll keep existing changes as is, but we'll split j9ddr_misc (for runtime/ddr/gcddr.cpp) and omrvmstartup (for omr/startup/omrvmstartup.cpp) into full and compressed, with the OVERRIDE flag set appropriately.


I'd like to see a change that would catch any other mis-matches: Ideas on how to do that, anyone?

I think it'd work if we swapped && !defined(OMR_OVERRIDE_COMPRESS_OBJECT_REFERENCES) for && !defined(OMR_MIXED_REFERENCES_MODE_STATIC), since OMR_MIXED_REFERENCES_MODE_STATIC is being seen by DDR (exists in superset and blob).

#if defined(OMR_GC_COMPRESSED_POINTERS) && defined(OMR_GC_FULL_POINTERS) && !defined(OMR_MIXED_REFERENCES_MODE_STATIC)
	bool _compressObjectReferences;
#endif /* defined(OMR_GC_COMPRESSED_POINTERS) && defined(OMR_GC_FULL_POINTERS) && !defined(OMR_MIXED_REFERENCES_MODE_STATIC) */

We'd have to update compressObjectReferences() to something like:

	MMINLINE bool compressObjectReferences() {
#if defined(OMR_GC_COMPRESSED_POINTERS)
#if defined(OMR_GC_FULL_POINTERS) && !defined(OMR_MIXED_REFERENCES_MODE_STATIC)
		return _compressObjectReferences;
#elif defined(OMR_GC_FULL_POINTERS) && defined(OMR_MIXED_REFERENCES_MODE_STATIC) && defined(OMR_OVERRIDE_COMPRESS_OBJECT_REFERENCES)
		return (bool)OMR_OVERRIDE_COMPRESS_OBJECT_REFERENCES;
#else /* defined(OMR_GC_FULL_POINTERS) && !defined(OMR_MIXED_REFERENCES_MODE_STATIC) */
		return true;
#endif /* defined(OMR_GC_FULL_POINTERS) && !defined(OMR_MIXED_REFERENCES_MODE_STATIC) */
#else /* defined(OMR_GC_COMPRESSED_POINTERS) */
		return false;
#endif /* defined(OMR_GC_COMPRESSED_POINTERS) */
	}

I think this would fix things without having to split the two libraries mentioned.

@keithc-ca
Copy link
Contributor

I don't understand how that fixes anything. It doesn't matter what preprocessor conditions we use, but we do need to have a consistent set of fields in all structures (at consistent offsets).

For gcddr.cpp, it only matters that the debug information matches reality (all the offsets are consistent with both regular and _full libraries - when they exist). The shared library j9ddr_misc is not included in any SDK.

omrvmstartup.cpp is another story. If we split the static library omrvmstartup, then doesn't that imply the need to split all things that link against it?

@sharon-wang
Copy link
Contributor Author

Using OMR_MIXED_REFERENCES_MODE_STATIC should make it so that the field doesn't get declared in any of the files currently declaring it, when we're building with mixed refs. My understanding is that since DDR doesn't know about OMR_OVERRIDE_COMPRESS_OBJECT_REFERENCES (!defined(OMR_OVERRIDE_COMPRESS_OBJECT_REFERENCES) ends up being true), the field gets declared, even though it shouldn't for mixed refs. Since DDR does know about OMR_MIXED_REFERENCES_MODE_STATIC, I believe the field wouldn't get included.

If we split the static library omrvmstartup, then doesn't that imply the need to split all things that link against it?

Yes, that is correct. So all the fvtest libraries and example would need to get split if we go that route.

@pshipton
Copy link
Member

Can't the field just always be included?

@sharon-wang
Copy link
Contributor Author

Can't the field just always be included?

We could, but clang on mac complains about the field being unused, since it doesn't get used in mixed refs builds. So we'd need to add a dummy function that "uses" it, or we can disable the -Wunused-private-field error for clang.

@gacholio
Copy link
Contributor

I'm leaning back towards doing this this with the dummy function or compiler warning disable for OSX. This means the field would always be there in the mixed builds no matter the override/splitting.

@gacholio
Copy link
Contributor

gacholio commented Jan 19, 2021

The error on mac talks about private fields - what if we just make all of the _compressObjectReferences fields public and initialize them in the constructors? Then we wouldn't need the function or warning supression.

@keithc-ca
Copy link
Contributor

I agree the field should always be present.

I also think it makes sense to have an accessor function (that would be used in code that isn't split and needs to query that state).

@gacholio
Copy link
Contributor

If a non-split class uses the affected header, it will get the existing accessor, which will query the field - there is no need for another function that no one will ever call.

@keithc-ca
Copy link
Contributor

If a non-split class uses the affected header, it will get the existing accessor, which will query the field - there is no need for another function that no one will ever call.

Fair enough. Then we just need to make the existence of such fields consistent and suppress the warning from xcode.

@gacholio
Copy link
Contributor

I'm hoping that making the fields public will avoid the warning.

@sharon-wang
Copy link
Contributor Author

Is there a preference to make the fields protected over public if that also works to avoid the warning?

@gacholio
Copy link
Contributor

I suppose protected is technically better, though I'm sure no one is actually worried about the visibility of the field. Please give it a try with protected.

@sharon-wang
Copy link
Contributor Author

Switching the field to protected is working for mac and it's looking good on BE platforms too. I'll open a PR with the changes.

@gacholio
Copy link
Contributor

To be clear, the field is now protected, and the !defined(override) has been removed from everywhere?

@sharon-wang
Copy link
Contributor Author

Yes, the field is protected in all the header files that declare it, and !defined(OMR_OVERRIDE_COMPRESS_OBJECT_REFERENCES) has been removed.

@sharon-wang
Copy link
Contributor Author

Opened: #11691 and eclipse-omr/omr#5758 with the fix.

Thank you everyone for your help investigating this!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment