Possible Memory leak in Apache Ignite?

I am trying to use Ignite as an in-memory database for key-value pairs with values ranging from 50MB to 800MB. It looks like Ignite allocates heap space through the JVM that it never cleans up, even if the cache entries are off heap, are cleared, there are no connected clients and no running operations. My machine would not be able to handle such memory consumption, hence I am looking for a way to clean up some memory.

My test scenario is as follows:

• Ignite version 2.9
• Running with OpenJDK 11

I am testing Ignite locally through a python script with the pyignite thin client:

client = Client()
client.connect('localhost', 10800)
my_cache = client.get_or_create_cache('default')
my_cache.clear()
data = createData() #creates 800 000 000 bytes test data
def put(caches):
i = caches
while i > 0:
my_cache.put('my key' + str(i), data)
i -= 1
put(5)
my_cache.remove_all()
my_cache.destroy()
client.close()

The script writes the 800 MB data sequentially in 5 different cache entries. The following snapshot illustrates how Ignite's Heap grows to its peak point, which by itself is understandable, but afterwards stays around 10GB even after performing GC:

Ignite heap

Performing a second test run with the same data does not result in bigger heap consumption after GC, which leads me to believe that Ignite allocates buffers for the incoming data internally, which correspond with the size of the data. This memory consumption is simply too great and my machine would not be able to deal with it in the long run.

The ignite configuration is quite simple:

<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd">
<bean id="ignite.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
<!-- Durable memory configuration. -->
<property name="dataStorageConfiguration">
<bean class="org.apache.ignite.configuration.DataStorageConfiguration">
<property name="dataRegionConfigurations">
<list>
<bean class="org.apache.ignite.configuration.DataRegionConfiguration">
<!-- Custom region name. -->
<property name="name" value="10GB_Region"/>
<!-- 100 MB initial size. -->
<property name="initialSize" value="#{100L * 1024 * 1024}"/>
<!-- 10GB maximum size. -->
<property name="maxSize" value="#{10096L * 1024 * 1024}"/>
</bean>
</list>
</property>
<!-- Redefining the default region's settings -->
<property name="defaultDataRegionConfiguration">
<bean class="org.apache.ignite.configuration.DataRegionConfiguration">
<property name="name" value="Default_Region"/>
<property name="maxSize" value="#{5L * 1024 * 1024 * 1024}"/>
</bean>
</property>
</bean>
</property>
<property name="cacheConfiguration">
<list>
<!-- Partitioned cache example configuration (Atomic mode). -->
<bean class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="default"/>
<property name="atomicityMode" value="ATOMIC"/>
<property name="backups" value="1"/>
<property name="dataRegionName" value="10GB_Region"/>
</bean>
</list>
</property>
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="ipFinder">
<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.multicast.TcpDiscoveryMulticastIpFinder">
<property name="addresses">
<list>
<value>127.0.0.1:47500..47509</value>
</list>
</property>
</bean>
</property>
</bean>
</property>
</bean>
</beans>

• ignite visor specifically says the the cache is off-heap
• VisualVM histogram says that 98% of live Bytes are mapped to byte[]
• it seems that there are client-connector and data-streamer-stripe threads that remain open, one per put operation in the cache, but I am not sure how much heap they require
• providing value hints in my_cache.put does not help
• cache.clear(), cache.destroy() does not help (and it should not, as the cache is off heap)

Any help will be very much appreciated!

1. Apache Ignite does not hold data on heap by default, instead, it will hold data off-heap.
2. Apahce Ignite will indeed not release off-heap when data is removed, but it will allow to re-use said off-heap when new data is ingested.
3. It's not clear at all why your tool would report the use of ~16G heap since Ignite should not use more than a few hundred MBs with the provided config. If that's not just heap but all RAM, then there you go. You should expect that 10G should be allocated. Why does it report more in Size field is not clear.

My apologies for the long delay to anyone who is interested, but I did get an answer to my question after contacting the Ignite mailing list. Apparently Ignite was never meant to consume such large amounts of data in a single query. The described behavior is expect and has to do with dividing the data into packages and sending it to other nodes or putting it into persistence. The suggested solution is to chunk the data into much smaller bits. Since this is currently not possible in my case, I had let go of Ignite and look for other approaches to store my data.