JEP 546: Adaptive Heap Sizing for ZGC

Summary

Enhance the Z Garbage Collector to adaptively adjust the size of the heap based on the resources available, the application's needs, and the needs of neighboring applications contending for the same resources.

Goals

• Adaptively adjust the size of the heap in response to changes in the workload and the host system.

• Provide the same throughput and latency as a well-tuned, manually configured heap.

• Ensure that existing command-line options for sizing and tuning the heap continue to work.

Motivation

The Java Virtual Machine presents the illusion of an infinite heap. The role of a garbage collector is to implement that illusion with the finite physical memory of the computer on which the JVM is running, using the facilities of the hardware and the operating system.

The Z Garbage Collector (ZGC) is a tracing garbage collector. To reclaim heap memory that is no longer needed by the application, it finds all of the in-use (i.e., live) objects by starting at roots, such as static fields of loaded classes and local variables in stack frames, and then following references from one object to another. It compacts the heap by copying live objects from old heap regions with few live objects into new regions, after which it frees the old regions. The CPU cost of a garbage collection cycle thus varies directly with the number and sizes of live objects. It is independent of the size of the heap, i.e., the amount of physical memory currently available for storing Java objects, whether live or not.

ZGC initiates a garbage collection cycle when it detects that the heap is nearly full. Since the CPU cost of a single collection cycle is independent of heap size, it follows that — all else being equal — a larger heap means lower GC CPU usage, over time, because collection cycles are less frequent. Conversely, a smaller heap means higher GC CPU usage, over time, because collection cycles are more frequent. In other words, GC CPU usage, over time, varies inversely with heap size.

Configuring the maximum heap size is difficult

In order to minimize its CPU usage, ZGC uses as much memory as it can, all the way up to the maximum heap size. By default, the HotSpot JVM uses 25% of the host system's physical memory as the maximum heap size. For example, on a system with 128GB of physical memory, the JVM will use 32GB for the heap.

While 25% of physical memory is a reasonable default, no specific maximum heap size is optimal for all applications. You can therefore specify the maximum heap size via the -Xmx option. Choosing a good maximum heap size requires a deep understanding of your application's behavior. This is best done by measuring your application's memory usage, throughput, and latency with different maximum heap sizes in an experimental environment with a representative workload. In practice, however, such experiments are challenging to get right and thus rarely done. To make matters worse, even when they are done, they must be redone each time the application or the JVM is updated. Their results are, moreover, not portable to other hardware configurations.

Choosing a bad maximum heap size can have adverse consequences. If the maximum heap size is too small, ZGC will use more CPU time to perform more frequent garbage collections. Performance will degrade, and the application might even run out of memory.

Configuring the JVM to be a good neighbor is difficult

If the maximum heap size is too large, other adverse consequences can arise: The operating system can run out of physical memory and resort to swapping some virtual memory pages out to disk, thereby degrading the performance of the entire system. In the worst case, the OS might kill processes in order to recover some physical memory.

The maximum heap size is a hard limit, which cannot be exceeded. You can configure ZGC to be a good neighbor to other processes on the host system by configuring a soft maximum heap size, which is smaller than the hard maximum heap size. ZGC strives to keep the heap size below the soft maximum by performing more frequent garbage collections, but it may expand the heap beyond the soft maximum, all the way to the hard maximum, if not doing so would stall an allocation or cause the application to run out of memory. (Allocation stalls are disastrous for the latency-sensitive applications that ZGC is intended to support.) The soft maximum allows ZGC to leave memory for other processes while still having a reserve of memory it can use if allocation rates spike unpredictably.

For example, consider an application that runs well with a 2GB heap but occasionally sees spikes in demand that require a 5GB heap. You could set the maximum heap size to 5GB (-Xmx5G), but that would risk impacting other processes. Alternatively, if you set the maximum heap size to 5GB and the soft maximum heap size to 2GB (-Xmx5G -XX:SoftMaxHeapSize=2G), ZGC will keep the heap under 2GB unless a workload spike occurs, in which case it will expand the heap to as much as 5GB and contract it back to 2GB after the spike ends.

Unfortunately, configuring the soft maximum heap size is no easier than configuring the hard maximum heap size, and choosing a bad soft maximum heap size can have the same kinds of adverse consequences.

Adaptive heap sizing

Configuring ZGC's hard and soft maximum heap sizes to achieve a good balance between memory usage and GC CPU usage is challenging. It would be far better if ZGC could automatically maintain a good balance on its own.

ZGC could do this by adaptively adjusting the size of the heap based on measured resource needs and availability, taking advantage of the fact that GC CPU usage tends to vary inversely with heap size. It could use more memory, when possible, in order to reduce GC CPU usage; likewise, it could use less memory, when needed, at the cost of increased GC CPU usage. This would reduce the need for manual GC tuning, better handle unexpected workload changes, be more robust across application and JVM updates, and be more portable across different hardware configurations. ZGC could, additionally, adaptively adjust the size of the heap in response to the amount of memory and CPU time available on the host system, so as to be a good neighbor to other applications.

Description

We propose an optional adaptive heap sizing policy for ZGC. When enabled, this policy will cause ZGC to

• Use a default maximum heap size of 100% of physical memory, minus a small reserve;

• Adaptively adjust the size of the heap in order to maintain a good balance between memory usage and GC CPU usage; and

• Act as a good neighbor by responding to changes in memory and CPU usage across the host system by expanding or contracting the heap, as though a soft maximum heap size had been set.

Together with associated improvements for startup and warmup, these changes will make it neither necessary nor desirable to set the maximum heap size (-Xmx) or the soft maximum heap size (-XX:SoftMaxHeapSize). When running your application, you need only choose ZGC and enable adaptive heap sizing:

$ java -XX:+UseZGC -XX:+ZAdaptiveHeapSizing ...

The case study below, of multiple applications running in a single container, shows ZGC adjusting the heap size and trading off CPU usage to ensure that all applications run equally well within the constraints of the container.

Tuning resource usage

For most applications, there is no need to tune ZGC's resource usage; the default will work well.

If needed, however, you can manually tune the balance of memory usage and GC CPU usage by specifying the target intensity of garbage collection via a command-line option. Intensity is a positive integer, with a default value of 5. Lower values make ZGC work less intensively: It initiates collection cycles less frequently, and thus uses less CPU time, at the cost of increasing the heap size, and thus using more physical memory. Higher values make ZGC work more intensively: It initiates collection cycles more frequently, and thus uses more CPU time, in order to reduce the heap size, and thus use less physical memory.

For example, to use a bit less CPU time but a bit more memory than the default setting:

$ java -XX:+UseZGC -XX:+ZAdaptiveHeapSizing -XX:ZGCIntensity=4 ...

Intensity is an abstract measure of GC CPU usage, meant to signal user intent. The relationship between intensity and actual GC CPU usage is intentionally unspecified. This will allow us to adjust the relationship in future releases as we improve ZGC's automatic tuning policies. An abstract measure of intensity has the additional benefit of being more portable across different hardware configurations.

A good steward of CPU and memory

The amount of heap memory in use by live objects changes continuously as the application creates objects and ZGC collects them. As the application warms up, ZGC aggressively expands the heap in order to stay ahead of the application's needs. Then, as the application runs, ZGC iteratively adjusts the size of the heap in order to achieve the target intensity:

• If GC CPU usage rises above the level indicated by the target intensity, ZGC expands the heap at the end of the next GC cycle. GC CPU usage will fall toward the target level and the application's throughput will improve.

• Conversely, if GC CPU usage drops below the level indicated by the target intensity, ZGC contracts the heap at the end of the next GC cycle. GC CPU usage will rise toward the target level and the application's throughput will decline.

Adjustments to the heap's size are smoothed out over time in order to avoid disruptive swings, as described by Tavakolisomeh et al. in their paper Heap Size Adjustment with CPU Control.

ZGC is a generational garbage collector. Newly created objects are placed in the young generation of the heap, while long-lived objects are promoted to the old generation. Frequent minor collections are performed on the young generation, while less frequent major collections are performed on the entire heap. In addition to expanding and contracting the heap, ZGC rebalances the sizes of the young and old generations in order to achieve the target intensity. If a series of minor collections causes GC CPU usage to rise above the level indicated by the target intensity, ZGC expands the heap immediately.

A good neighbor

If we let ZGC use an arbitrary amount of memory, or an arbitrary amount of CPU time, the host system may not have enough memory or CPU time available to run other processes. Therefore, in addition to monitoring the behavior of the application, ZGC monitors the fractions of the host system's memory and CPU time used by the JVM process and reacts accordingly:

• If the JVM process is using a significant fraction of the host system's memory but a small fraction of the CPU time, ZGC contracts the heap, thereby increasing its CPU usage.

• Conversely, if the JVM process is using a small fraction of the host system's memory but a large fraction of the CPU time, ZGC expands the heap, thereby decreasing its CPU usage.

ZGC prioritizes being a good neighbor over achieving its target intensity. It does so by biasing the default or specified target intensity up or down, as appropriate. In this way, multiple JVMs using ZGC on the same host system will use memory and CPU time in a manner consistent with their respective target intensities.

Case study in adaptive heap sizing

In this case study, we create a container with 16GB of memory. In that container, we start an instance of the SPECjbb2015 benchmark application, and then an H2 in-memory database benchmark application. When the database benchmark finishes, we start two more instances of SPECjbb2015. We monitor the physical memory usage of the container and of the four JVMs for 42 minutes.

During the first 12 minutes, the initial instance of SPECjbb2015 processes 5,000 requests per second. ZGC finds a reasonable heap size that yields good performance while fitting within the container's limits. The heap expands slowly, since ZGC's CPU usage is not excessive.

In the second 12 minutes, the H2 database application runs in parallel with the initial SPECjbb2015 instance. The database JVM expands its heap in a burst early on; in response, the SPECjbb2015 JVM immediately contracts its heap, allowing the memory-intensive database JVM to claim a larger share of the available memory. Throughout this period, the SPECjbb2015 instance continues to process 5,000 requests per second without saturating the CPU. Its GC CPU usage is higher, however, thereby degrading both latency and throughput.

After the database application exits, at the 24-minute mark, the initial SPECjbb2015 JVM expands its heap to use the memory previously used by the database. ZGC expands the heap rapidly in order to reduce its CPU usage, thereby improving both latency and throughput.

In the last phase, starting at the 30-minute mark, two additional SPECjbb2015 instances run in parallel with the initial SPECjbb2015 instance, each processing 5,000 requests per second. Within two minutes, the three JVMs have evenly divided the available memory, ensuring good performance for each.

Alternatives

Garbage collectors in some other language implementations allow you to control the balance between memory usage and GC CPU usage by setting a target memory residency, i.e., a target ratio of the amount of live data to the total heap size. For a given residency, however, the GC's CPU usage can vary dramatically. Measuring the amount of live data is, moreover, challenging in generational GCs. Measuring the GC's CPU usage is not only more straightforward, but it more directly represents what users monitor and care about.

Risks and Assumptions

When adaptive heap sizing is enabled, ZGC will use a default maximum heap size of 100% of physical memory, minus a small reserve, rather than the original default of 25% of physical memory. There is thus a risk that, for a given application, ZGC will use more memory than it did in past releases, leaving other processes with insufficient memory. Even with the original default maximum heap size, however, leaving other processes with insufficient memory was already a risk when several JVMs using that default ran in parallel.