The java.lang.OutOfMemoryError: Metaspace indicates that allocated native memory for Java class metadata is exausted. That’s why issue occured in standalone and cloud based applications.
In Java 8 and later versions, the maximum amount of memory allocated for Java classes (MaxMetaspaceSize) is by default unlimited, so in most cases there is no need to change this setting. On the other hand, if you want to fix the amount of memory allocated for Java classes, you can set it as follows:
This JVM parameter -XX:MaxMetaspaceSize is just an set the upper limit of MetaSpace. The current Metaspace size (i.e. committed) will be smaller. In fact, there is a setting called MaxMetaspaceFreeRatio (default 70%) which means that the actual metaspace size will never exceed 230% of its occupancy.
And for it to grow it first would have to fill up, forcing a garbage collection in an attempt to free objects and only when it cannot meet its MinMetaspaceFreeRatio (default 40%) goal it would expand the current metaspace. That can however not be greater than 230% of the occupancy after the GC cycle.
Monitoring MetaSpace Size with Java Native Memory tracking
A good way to monitor the exact amount of Metadata is by using the NativeMemoryTracking, which can be added through the following settings:
-XX:+UnlockDiagnosticVMOptions -XX:NativeMemoryTracking=detail -XX:+PrintNMTStatistics
When Native Memory Tracking is enabled, you can request a report on the JVM memory usage using the following command:
$ jcmd <pid> VM.native_memory
OutOfMemoryError: Metaspace on OpenShift/Kubernetes
When using openjdk Image on OpenShift/Kubernetes, the default maxium value for the Metaspace is XX:MaxMetaspaceSize=100m. You might have noticed that setting this value through the JAVA_OPTIONS environment variable, doesn’t work as the default value is appended to the bottom:
VM Arguments: -Xms128m -Xmx1024m -XX:MetaspaceSize=128M -XX:MaxMetaspaceSize=256m -XX:AdaptiveSizePolicyWeight=90 -XX:MaxMetaspaceSize=100m -XX:+ExitOnOutOfMemoryError
Once the MetaSpace get full in your application it will stop the service and through exception as below in your logs.
oc logs XYZ-service-7b856cc89-kpc6k | grep -i metaspace INFO exec java -javaagent:/usr/share/java/jolokia-jvm-agent/jolokia-jvm.jar=config=/opt/jboss/container/jolokia/etc/jolokia.properties -XX:+UseParallelOldGC -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -XX:MaxMetaspaceSize=100m -XX:+ExitOnOutOfMemoryError -cp "." -jar /deployments/XYZ-service-0.0.1-SNAPSHOT.jar Picked up JAVA_TOOL_OPTIONS: -Dappdynamics.agent.accountAccessKey=600a90af-582a-4ae2-87b1-4599708b65dd -Dappdynamics.agent.reuse.nodeName=true -Dappdynamics.socket.collection.bci.enable=true -XX:MaxMetaspaceSize=1024m -javaagent:/opt/appdynamics-java/javaagent.jar Terminating due to java.lang.OutOfMemoryError: Metaspace
The correct way to set the MaxMetaspaceSize is through the GC_MAX_METASPACE_SIZE environment variable. Here are the different cases ti implement this solutions:
- Jenkins Pipeline: For example, if you are using a jenkins pipeline to deploy your application or services then you can meke following changes in the json template to refelect these changes in deployment.yaml file to deploy your application with JKube, the following settings will override the default values for the MaxMetaspaceSize and MaxMetaspaceSize:
spec: template: spec: containers: - env: - name: JAVA_OPTIONS value: '-Xms256m -Xmx1024m' - name: GC_MAX_METASPACE_SIZE value: 1024 - name: GC_METASPACE_SIZE value: 256
- Manual Deployment through S2I: You can directly pass these MetaSpace parameters while deploying your service manually.
oc new-app xyz-service -e JAVA_OPTIONS="-Xms256m -Xmx1024m" -e GC_MAX_METASPACE_SIZE=1024 -e GC_METASPACE_SIZE=256
Note : After deploying your service over OpenShift/Kernates validate the deployment configuration file(deployment.yml) for these parameters. In case not not reflecting then delete your pods completly and reploy the application.
In case, you want make changes on other parameters also for improving the puformance of your applictaion then you can follow these documents to list of parameters for OpenShift.
Let me know your thought on this post. if this solution was helpful for you make comment on the post.
Happy Learning !!!