Memory Leak from LazyMOMProvider

my java program is eating a bit too much memory suddenly, and so i dumped its contents while it was in a high usage state and analysed it in Eclipse Memory Analyzer and i get that LazyMOMProvider through endpointsWaitingForMOM is eating up 90% of its memory to an amount of 2.7Gb:

Now i did some analyzing and i DO NOT use that class ever in my code, maybe it's getting called by some dependency i use that has a dependency on this class itself, but do not understand at all what is happening here.

Anyone have had the same issue that can help me out?

This class comes from glassfish App server which is very hard to dig in or predict until getting my hands dirty to the server and source code which is impossible, since you yourself don't know what causes the issue.

public enum LazyMOMProvider {

INSTANCE;

/**
 * Possible scopes (environments) in which the provider (and object associated with it) could be in.
 * Default scope is STANDALONE - applied during initialization of classes. For now, only possible scope change for a
 * object can be in this direction: STANDALONE -> GLASSFISH_NO_JMX -> GLASSFISH_JMX.
 */
public static enum Scope {

    //** Default scope where lazy flag is not applied and all Gmbal API calls are processed immediately. */
    STANDALONE,

    /** In this scope almost all Gmbal API call are deferred until a JMX connection to a Glassfish server is created */
    GLASSFISH_NO_JMX,

    /** Same as STANDALONE. Gmbal API calls are processed immediately. */
    GLASSFISH_JMX

}

/**
 * Interface for all object that want to be notified about scope change, introducing required methods.
 */
public static interface ScopeChangeListener {

    void scopeChanged(Scope scope);

}

/**
 * Default interface for all object that want to be notified about scope change. This interface should not be
 * implemented directly.
 */
public static interface DefaultScopeChangeListener extends ScopeChangeListener {
}

/**
 * Interface used for distinguishing between a registration of a WSEndpointImpl rather than of other classes.
 * Webservice Endpoints should get a notification about scope change sooner than the rest of the registered listeners
 * (there is a possibility that other listeners are dependant on Webservice Endpoints).
 */
public static interface WSEndpointScopeChangeListener extends ScopeChangeListener {
}

private final Set<WSEndpointScopeChangeListener> endpointsWaitingForMOM = new HashSet();
private final Set<DefaultScopeChangeListener> listeners = new HashSet();

private volatile Scope scope = Scope.STANDALONE;

/**
 * Initializes this provider with a given scope. If the given scope is different than the one this provider is
 * currently in and the transition between scopes is valid then a event is fired to all registered listeners.
 *
 * @param scope a scope to initialize this provider with
 */
public void initMOMForScope(LazyMOMProvider.Scope scope) {
    // cannot go backwards between scopes, for possible scope changes see #Scope
    if ((this.scope == Scope.GLASSFISH_JMX)
            || (scope == Scope.STANDALONE && (this.scope == Scope.GLASSFISH_JMX || this.scope == Scope.GLASSFISH_NO_JMX))
            || this.scope == scope) {
        return;
    }

    this.scope = scope;

    fireScopeChanged();
}

/**
 * Notifies the registered listeners about the scope change.
 */
private void fireScopeChanged() {
    for (ScopeChangeListener wsEndpoint : endpointsWaitingForMOM) {
        wsEndpoint.scopeChanged(this.scope);
    }

    for (ScopeChangeListener listener : listeners) {
        listener.scopeChanged(this.scope);
    }
}

/**
 * Registers the given object as a listener.
 *
 * @param listener a listener to be registered
 */
public void registerListener(DefaultScopeChangeListener listener) {
    listeners.add(listener);

    if (!isProviderInDefaultScope()) {
        listener.scopeChanged(this.scope);
    }
}

/**
 * Returns {@code true} if this provider is in the default scope.
 *
 * @return {@code true} if this provider is in the default scope,
 *          {@code false} otherwise
 */
private boolean isProviderInDefaultScope() {
    return this.scope == Scope.STANDALONE;
}

public Scope getScope() {
    return scope;
}

/**
 * Registers a Webservice Endpoint as a listener.
 * Webservice Endpoints should rather register through this method than through LazyMOMProvider#registerListener
 * because generally they need to be notified sooner than arbitrary listener (i.e. object that is dependant on
 * Webservice Endpoint)
 *
 * @param wsEndpoint a Webservice Endpoint to be registered
 */
public void registerEndpoint(WSEndpointScopeChangeListener wsEndpoint) {
    endpointsWaitingForMOM.add(wsEndpoint);

    if (!isProviderInDefaultScope()) {
        wsEndpoint.scopeChanged(this.scope);
    }
}

/**
 * Unregisters a Webservice Endpoint from the list of listeners.
 *
 * @param wsEndpoint a Webservice Endpoint to be unregistered
 */
public void unregisterEndpoint(WSEndpointScopeChangeListener wsEndpoint) {
    endpointsWaitingForMOM.remove(wsEndpoint);
}

}

Source Code: LazyMOMProvider.java

My quick troubleshooting advice is to try the following in testbed

1. you must be using java 8 which you can upgrade to Java 11 or 17
2. Try moving your app out from glassfish and migrate it to better server like IBM WebSphere or Oracle Weblogic or Oracle App server, etc and see your application's performance.
3. In the worst case, Try upgrading your glassfish server and try running the app if you have no command over upgrading Java or server replacements.

It might take a day. but Trust me It helps.

It looks like your application somehow registers some endpoints which ends in endpointsWaitingForMOM HashSet and never un-registers them --> memory leak, so you need somehow call that unregisterEndpoint method.

Have a look at the Javadoc of the class - there is some description on how to use it in standalone applications - maybe this shines some light on the solution.

Even though illogic in my opinion, this situation stemmed from a WebService we had being reset in the middle of the night.

Apparently the WebService was being reset every 30 seconds between 1 and 3 AM, the reset method doesn't work as it should, because it keeps the objects alive and they are not GC'd.

By using a different method of stopping the WebService the objects are properly killed and this situation is gone.