refactor: replace -h with value provided by MemoryMXBean

HyperAlloc/README.md

@@ -15,7 +15,7 @@ To configure HyperAlloc to reach a high sustained allocation rate, there are two
 
 As an experimental feature, HyperAlloc makes dynamic changes to the created object graph in order to exercise the memory read and write barriers typical of concurrent garbage collectors. Before beginning its main test phase, it stores long-lived objects in a hierarchical list of object groups. In order to exercise garbage collector marking phases, higher group objects randomly reference objects in the next lower group to create a somewhat complex and randomized reference graph. This graph does not remain static: HyperAlloc constantly replaces a portion of it and reshuffles references between the objects in it. You can control the long-lived object replacement ratio by specifying the -r option (<ratio of objects being replaced per minute>, default: 50). The default value means that 1/50 of objects will be replaced per minute. The reshuffled object reference ratio (-f <ratio of objects get reshuffled>, default: 100) default value means that when replacement happens, 1/100 of inter-object references are reshuffled.
 
-To predict heap occupancy and allocation rates, HyperAlloc makes its own calculations based on knowledge of JVM-internal object representations, which depend on the JVM implementation in use. These are currently specific to the HotSpot JVM for JDK 8 or later. The calculations seem to agree with what HotSpot GC logs indicate as long as the following parameter is used correctly. HyperAlloc cannot automatically detect when the JVM uses compressed object references, i.e., 32-bit object references in a 64-bit JVM, aka “compressedOops”. You need to set the parameter “-c” to false when running HyperAlloc with a 32 GB or larger heap or with a collector that does not support “compressedOops”.
+To predict heap occupancy and allocation rates, HyperAlloc makes its own calculations based on knowledge of JVM-internal object representations, which depend on the JVM implementation in use. These are currently specific to the HotSpot JVM for JDK 8 or later. The calculations seem to agree with what HotSpot GC logs indicate as long as the following parameter is used correctly. HyperAlloc automatically detects when the JVM uses compressed object references, i.e., 32-bit object references in a 64-bit JVM, aka “compressedOops”.
 
 HyperAlloc, while written from scratch, inherits its basic ideas from Gil Tene’s [HeapFragger](https://github.com/giltene/HeapFragger) workload. HeapFragger has additional features (e.g., inducing fragmentation and detecting generational promotion), whereas HyperAlloc concentrates on accurately predicting the resulting allocation rate. Additionally, we thank to Gil for his [jHiccup](https://www.azul.com/jhiccup/) agent, which we utilize to measure JVM pauses.
 
@@ -38,7 +38,7 @@ If you would like to report a potential security issue in this project, please d
 
 Invocation with the minimum recommended set of HyperAlloc parameters and a typical jHiccup configuration:
 ```
-java -Xmx<bytes> -Xms<bytes> <GC options> <other JVM options> -Xloggc:<GC log file> -javaagent:<jHiccup directory>/jHiccup.jar='-a -d 0 -i 1000 -l <jHiccup log file>' -jar <HyperAlloc directory>/HyperAlloc-1.0.jar -a <MB> -h <MB> -d <seconds> -c <true/false> -l <CVS output file>
+java -Xmx<bytes> -Xms<bytes> <GC options> <other JVM options> -Xloggc:<GC log file> -javaagent:<jHiccup directory>/jHiccup.jar='-a -d 0 -i 1000 -l <jHiccup log file>' -jar <HyperAlloc directory>/HyperAlloc-1.0.jar -a <MB> -d <seconds> -l <CVS output file>
 ```
 
 ### Build
@@ -54,10 +54,7 @@ The JAR file can be found in the *target* folder.
 The two primary arguments are allocation rate and heap occupancy:
 
 * -a < target allocation rate in Mb per second >, default: 1024
-* -s < target heap occupancy in Mb >, default: 64 
-
-Currently, the benchmark program needs to be told the heap size in use.
-* -h < heap size in Mb >
+* -s < target heap occupancy in Mb >, default: 64
 
 The benchmark cannot always achieve the specified values. In particular, the run duration must be long enough for HyperAlloc to meet the heap occupancy target, especially for those low allocation rate cases. You can set the benchmark run duration using:
 
@@ -67,10 +64,6 @@ At end of the run, HyperAlloc writes the actual achieved allocation rate and the
 
 * -l < result file name >, default: output.csv
 
-If you run with a 32G or larger heap or with a collector that does not support 32-bit object pointers, aka "compressedOops", you must set this paramteter to "false". Otherwise all object size calculations are off by nearly 50%. Currently, HyperAlloc does not automatically detect this.
-
-* -c < compressedOops support >, default: true
-
 In order to achieve high allocation rates, HyperAlloc uses multiple worker threads. If the hardware has enough CPU cores, you can increase the number of worker threads to ensure achieving the target allocation rate.
 
 * -t < number of worker threads >, default: 4
@@ -111,7 +104,7 @@ In order to calm the allocation rate during startup, HyperAlloc can gradually in
 We normally use [JHiccup](https://www.azul.com/jhiccup/) to measure JVM pauses. You can either download it from its [website](https://www.azul.com/jhiccup-2/), or build it from the source code in its [GitHub repo](https://github.com/giltene/jHiccup). You can also use GC logs to measure safepoint times, allocation stalls, and Garbage Collection pauses. In the example below, we run HyperAlloc for the Shenandoah collector for 10 minutes using a 16Gb/s allocation rate and with 32Gb of a 64Gb heap occupied by long-lived objects.
 
 ```
-jdk/jdk-13.0.2+8/bin/java -Xmx65536m -Xms65536m -XX:+UnlockExperimentalVMOptions -XX:+UseShenandoahGC -XX:+UseLargePages -XX:+AlwaysPreTouch -XX:-UseBiasedLocking -Xloggc:./results/16384_65536_32768/gc.log -javaagent:<path to jHiccup>/jHiccup.jar='-a -d 0 -i 1000 -l ./results/16384_65536_32768/hyperalloc.hlog' -jar ./buildRoot/jar/HyperAlloc-1.0.jar -a 16384 -h 32768 -d 600 -m 128 -c false -t 16 -n 64 -x 32768 -l ./results/16384_65536_32768/output.csv
+jdk/jdk-13.0.2+8/bin/java -Xmx65536m -Xms65536m -XX:+UnlockExperimentalVMOptions -XX:+UseShenandoahGC -XX:+UseLargePages -XX:+AlwaysPreTouch -XX:-UseBiasedLocking -Xloggc:./results/16384_65536_32768/gc.log -javaagent:<path to jHiccup>/jHiccup.jar='-a -d 0 -i 1000 -l ./results/16384_65536_32768/hyperalloc.hlog' -jar ./buildRoot/jar/HyperAlloc-1.0.jar -a 16384 -d 600 -m 128 -c false -t 16 -n 64 -x 32768 -l ./results/16384_65536_32768/output.csv
 ```
 
 This command sets JHiccup as a Java agent and use it to create the hiccup log. The *output.csv* file contains the following information:

HyperAlloc/src/main/java/com/amazon/corretto/benchmark/hyperalloc/HyperAlloc.java

@@ -8,6 +8,7 @@ public final class HyperAlloc {
     private HyperAlloc() {}
 
     public static void main(String[] args) {
+
         switch (findRunType(args)) {
             case "simple" :
                 new SimpleRunner(new SimpleRunConfig(args)).start();

HyperAlloc/src/main/java/com/amazon/corretto/benchmark/hyperalloc/RunReport.java

@@ -0,0 +1,40 @@
+package com.amazon.corretto.benchmark.hyperalloc;
+
+import java.io.Closeable;
+import java.io.FileWriter;
+import java.io.IOException;
+
+public class RunReport implements Closeable {
+
+    private FileWriter fw;
+
+    public RunReport(String path) throws IOException {
+        fw = new FileWriter(path,true);
+    }
+
+    public void write(String message) throws IOException {
+        fw.write(message);
+        fw.write(",");
+    }
+
+    public void write(int value) throws IOException {
+        this.write(Integer.toString(value));
+    }
+
+    public void write(long value) throws IOException {
+        this.write(Long.toString(value));
+    }
+
+    public void eol() throws IOException {
+        fw.write("\n");
+    }
+
+    public void write(boolean value) throws IOException {
+        this.write(Boolean.toString(value));
+    }
+
+    @Override
+    public void close() throws IOException {
+        fw.close();
+    }
+}

HyperAlloc/src/main/java/com/amazon/corretto/benchmark/hyperalloc/SimpleRunConfig.java

@@ -2,6 +2,10 @@
 // SPDX-License-Identifier: Apache-2.0
 package com.amazon.corretto.benchmark.hyperalloc;
 
+import com.sun.management.HotSpotDiagnosticMXBean;
+
+import java.lang.management.ManagementFactory;
+
 /**
  * Class for parsing simple run parameters.
  */
@@ -13,15 +17,20 @@ public class SimpleRunConfig {
     private int numOfThreads = 4;
     private int minObjectSize = 128;
     private int maxObjectSize = 1024;
-    private boolean useCompressedOops = true;
+    private boolean useCompressedOops;
     private int pruneRatio = ObjectStore.DEFAULT_PRUNE_RATIO;
     private int reshuffleRatio = ObjectStore.DEFAULT_RESHUFFLE_RATIO;
-    private int heapSizeInMb = 1024;
+    private int heapSizeInMb = (int)(ManagementFactory.getMemoryMXBean().getHeapMemoryUsage().getMax() / 1048576L);
     private String logFile = "output.csv";
     private String allocationLogFile = null;
     private Double allocationSmoothnessFactor = null;
     private double rampUpSeconds = 0.0;
 
+    {
+        HotSpotDiagnosticMXBean mxBeanServer = ManagementFactory.getPlatformMXBean(HotSpotDiagnosticMXBean.class);
+        useCompressedOops = Boolean.parseBoolean(mxBeanServer.getVMOption("UseCompressedOops").getValue());
+    }
+
     /**
      * Parse input arguments from a string array.
      * @param args The string array of the arguments.
@@ -31,7 +40,9 @@ public SimpleRunConfig(final String[] args) {
             if (args[i].equals("-a")) {
                 allocRateInMbPerSecond = Long.parseLong(args[++i]);
             } else if (args[i].equals("-h")) {
-                heapSizeInMb = Integer.parseInt(args[++i]);
+                ++i;
+                // Left in to be compatible with existing scripts
+                System.out.println("Use of -h has been deprecated - using value retrieved from MemoryMXBean");
             } else if (args[i].equals("-s")) {
                 longLivedInMb = Integer.parseInt(args[++i]);
             } else if (args[i].equals("-m")) {
@@ -49,7 +60,8 @@ public SimpleRunConfig(final String[] args) {
             } else if (args[i].equals("-f")) {
                 reshuffleRatio = Integer.parseInt(args[++i]);
             } else if (args[i].equals("-c")) {
-                useCompressedOops = Boolean.parseBoolean(args[++i]);
+                ++i;
+                System.out.println("Use of -c has been deprecated - using value retrieved from HotSpotDiagnosticMXBean");
             } else if (args[i].equals("-z")) {
                 allocationSmoothnessFactor = Double.parseDouble(args[++i]);
                 if (allocationSmoothnessFactor < 0 || allocationSmoothnessFactor > 1.0) {
@@ -73,9 +85,9 @@ public SimpleRunConfig(final String[] args) {
 
     private void usage() {
         System.out.println("Usage: java -jar HyperAlloc.jar " +
-                "[-u run type] [-a allocRateInMb] [-h heapSizeInMb] [-s longLivedObjectsInMb] " +
+                "[-u run type] [-a allocRateInMb] [-s longLivedObjectsInMb] " +
                 "[-m midAgedObjectsInMb] [-d runDurationInSeconds ] [-t numOfThreads] [-n minObjectSize] " +
-                "[-x maxObjectSize] [-r pruneRatio] [-f reshuffleRatio] [-c useCompressedOops] " +
+                "[-x maxObjectSize] [-r pruneRatio] [-f reshuffleRatio] " +
                 "[-l outputFile] [-b|-allocation-log logFile] [-z allocationSmoothness (0 to 1.0)] " +
                 "[-p rampUpSeconds ]");
     }
@@ -105,7 +117,6 @@ public SimpleRunConfig(final long allocRateInMbPerSecond, final double allocSmoo
                            final String allocationLogFile, final double rampUpSeconds) {
         this.allocRateInMbPerSecond = allocRateInMbPerSecond;
         this.allocationSmoothnessFactor = allocSmoothnessFactor;
-        this.heapSizeInMb = heapSizeInMb;
         this.longLivedInMb = longLivedInMb;
         this.midAgedInMb = midAgedInMb;
         this.durationInSecond = durationInSecond;
@@ -114,7 +125,6 @@ public SimpleRunConfig(final long allocRateInMbPerSecond, final double allocSmoo
         this.maxObjectSize = maxObjectSize;
         this.pruneRatio = pruneRatio;
         this.reshuffleRatio = reshuffleRatio;
-        this.useCompressedOops = useCompressedOops;
         this.logFile = logFile;
         this.allocationLogFile = allocationLogFile;
         this.rampUpSeconds = rampUpSeconds;

HyperAlloc/src/main/java/com/amazon/corretto/benchmark/hyperalloc/SimpleRunner.java

@@ -27,6 +27,7 @@ public class SimpleRunner extends TaskBase {
     private static final AtomicInteger THREAD_COUNTER = new AtomicInteger(0);
 
     private final SimpleRunConfig config;
+    private long realAllocRate;
 
     public SimpleRunner(SimpleRunConfig config) {
         this.config = config;
@@ -36,6 +37,7 @@ public SimpleRunner(SimpleRunConfig config) {
 
     @Override
     public void start() {
+        System.out.println("Starting a SimpleRunner");
         try {
             AllocObject.setOverhead(config.isUseCompressedOops() ? AllocObject.ObjectOverhead.CompressedOops
                     : AllocObject.ObjectOverhead.NonCompressedOops);
@@ -64,7 +66,6 @@ public void start() {
                 }
             } catch (ExecutionException ex) {
                 logger.log(Level.SEVERE, ex.getMessage(), ex);
-                printResult(-1);
                 System.exit(1);
             }
 
@@ -87,29 +88,16 @@ public void start() {
                 Thread.currentThread().interrupt();
             }
             store.stopAndReturnSize();
-            printResult(AllocObject.getBytesAllocated() / 1024 / 1024 / config.getDurationInSecond());
+            realAllocRate = AllocObject.getBytesAllocated() / 1024 / 1024 / config.getDurationInSecond();
+            try (RunReport report = new RunReport(config.getLogFile())) {
+                this.writeOn(report);
+            }
         } catch (Exception ex) {
             ex.printStackTrace();
             System.exit(1);
         }
     }
 
-    private void printResult(final long realAllocRate) throws IOException {
-        try (FileWriter fw = new FileWriter(config.getLogFile(), true)) {
-            fw.write(config.getHeapSizeInMb() + ","
-                    + config.getAllocRateInMbPerSecond() + ","
-                    + realAllocRate + ","
-                    + ((double) (config.getLongLivedInMb() + config.getMidAgedInMb()) / config.getHeapSizeInMb()) + ","
-                    + config.isUseCompressedOops() + ","
-                    + config.getNumOfThreads() + ","
-                    + config.getMinObjectSize() + ","
-                    + config.getMaxObjectSize() + ","
-                    + config.getPruneRatio() + ","
-                    + config.getReshuffleRatio() + ",\n"
-            );
-        }
-    }
-
     private List<Callable<Long>> createTasks(final ObjectStore store) {
         final int queueSize = (int) (config.getMidAgedInMb() * 1024L * 1024L * 2L
                 / (config.getMaxObjectSize() + config.getMinObjectSize())
@@ -204,4 +192,18 @@ public void run() {
             }
         }
     }
+
+    public void writeOn(RunReport report) throws IOException {
+        report.write(config.getHeapSizeInMb());
+        report.write(config.getAllocRateInMbPerSecond());
+        report.write(realAllocRate);
+        report.write((config.getLongLivedInMb() + config.getMidAgedInMb()) / config.getHeapSizeInMb());
+        report.write(config.isUseCompressedOops());
+        report.write(config.getNumOfThreads());
+        report.write(config.getMinObjectSize());
+        report.write(config.getMaxObjectSize());
+        report.write(config.getPruneRatio());
+        report.write(config.getReshuffleRatio());
+        report.eol();
+    }
 }

HyperAlloc/src/main/java/com/amazon/corretto/benchmark/hyperalloc/TaskBase.java

@@ -2,6 +2,7 @@
 // SPDX-License-Identifier: Apache-2.0
 package com.amazon.corretto.benchmark.hyperalloc;
 
+import java.io.IOException;
 import java.util.ArrayDeque;
 import java.util.concurrent.Callable;
 import java.util.concurrent.TimeUnit;

HyperAlloc/src/test/java/com/amazon/corretto/benchmark/hyperalloc/SimpleRunConfigTest.java

@@ -2,14 +2,29 @@
 // SPDX-License-Identifier: Apache-2.0
 package com.amazon.corretto.benchmark.hyperalloc;
 
+import com.sun.management.HotSpotDiagnosticMXBean;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
 import org.junit.jupiter.api.Test;
 
+import java.lang.management.ManagementFactory;
+
 import static com.github.stefanbirkner.systemlambda.SystemLambda.catchSystemExit;
 import static org.hamcrest.MatcherAssert.assertThat;
 import static org.hamcrest.CoreMatchers.is;
 import static org.junit.jupiter.api.Assertions.*;
 
 class SimpleRunConfigTest {
+
+    // value will change depending on how much memory the test machine has.
+    int maxHeap = (int)(ManagementFactory.getMemoryMXBean().getHeapMemoryUsage().getMax() / 1048576L);
+    boolean useCompressedOops;
+
+    {
+        HotSpotDiagnosticMXBean mxBeanServer = ManagementFactory.getPlatformMXBean(HotSpotDiagnosticMXBean.class);
+        useCompressedOops = Boolean.parseBoolean(mxBeanServer.getVMOption("UseCompressedOops").getValue());
+    }
+
     @Test
     void DefaultStringsTest() {
         final SimpleRunConfig config = new SimpleRunConfig(new String[0]);
@@ -19,7 +34,7 @@ void DefaultStringsTest() {
         assertThat(config.getDurationInSecond(), is(60));
         assertThat(config.getMaxObjectSize(), is(1024));
         assertThat(config.getMinObjectSize(), is(128));
-        assertThat(config.getHeapSizeInMb(), is(1024));
+        assertThat(config.getHeapSizeInMb(), is(maxHeap));
         assertThat(config.getLongLivedInMb(), is(64));
         assertThat(config.getMidAgedInMb(), is(64));
         assertThat(config.getPruneRatio(), is(50));
@@ -39,13 +54,13 @@ void ConstructorTest() {
         assertThat(config.getDurationInSecond(), is(3000));
         assertThat(config.getMaxObjectSize(), is(512));
         assertThat(config.getMinObjectSize(), is(256));
-        assertThat(config.getHeapSizeInMb(), is(32768));
+        assertThat(config.getHeapSizeInMb(), is(maxHeap));
         assertThat(config.getLongLivedInMb(), is(256));
         assertThat(config.getMidAgedInMb(), is(32));
         assertThat(config.getPruneRatio(), is(10));
         assertThat(config.getReshuffleRatio(), is(20));
         assertThat(config.getLogFile(), is("nosuch.csv"));
-        assertFalse(config.isUseCompressedOops());
+        assertEquals(useCompressedOops,config.isUseCompressedOops());
     }
 
     @Test
@@ -54,26 +69,26 @@ void StringArgsTest() {
                                                                         "-d", "3000", "-m", "32", "-t", "16",
                                                                         "-f", "20", "-r", "10", "-x", "512", "-u", "simple",
                                                                         "-n", "256", "-c", "false", "-l", "nosuch.csv"});
-
         assertThat(config.getNumOfThreads(), is(16));
         assertThat(config.getAllocRateInMbPerSecond(), is(16384L));
         assertThat(config.getDurationInSecond(), is(3000));
         assertThat(config.getMaxObjectSize(), is(512));
         assertThat(config.getMinObjectSize(), is(256));
-        assertThat(config.getHeapSizeInMb(), is(32768));
+        assertThat(config.getHeapSizeInMb(), is(maxHeap));
         assertThat(config.getLongLivedInMb(), is(256));
         assertThat(config.getMidAgedInMb(), is(32));
         assertThat(config.getPruneRatio(), is(10));
         assertThat(config.getReshuffleRatio(), is(20));
         assertThat(config.getLogFile(), is("nosuch.csv"));
-        assertFalse(config.isUseCompressedOops());
+        assertEquals(useCompressedOops,config.isUseCompressedOops());
     }
 
     @Test
     void UnknownParameterShouldExitTest() throws Exception {
         int status = catchSystemExit(
                 () -> new SimpleRunConfig(new String[]{"-w", "who"}));
-
         assertThat(status, is(1));
     }
+
+    class MySecurityManager extends SecurityManager {}
 }