英文:
When to close DL4J INDArrays
问题
我创建了一个自定义的DataSetIterator。它通过在next方法中随机生成两个INDArray（一个用于输入，一个用于输出），然后创建一个DataSet：
int[][] inputArray = new int[num][NUM_INPUTS];
int[][] expectedOutputArray = new int[num][];
for (int i = 0; i < num; i++) {//just fill the arrays with some data
int sum = 0;
int product = 1;
for (int j = 0; j < inputArray[i].length; j++) {
inputArray[i][j] = rand.nextInt();
sum += inputArray[i][j];
product *= inputArray[i][j];
}
expectedOutputArray[i] = new int[] { sum, product, sum / inputArray[i].length };
}
INDArray inputs = Nd4j.createFromArray(inputArray);//never closed
INDArray desiredOutputs = Nd4j.createFromArray(expectedOutputArray);//never closed
return new DataSet(inputs, desiredOutputs);
然而，INDArray 实现了 AutoClosable 接口，其 close() 方法的文档 表示：
> 此方法释放此 INDArray 实例使用的独占的 off-heap 资源。如果 INDArray 依赖于共享资源，将抛出异常。请注意：此方法根本不安全。
-
我需要关闭这些
INDArray吗？ -
如果需要，我什么时候需要关闭这些
INDArray？
我已经尝试使用了 try-with-resources，但在 fit 方法中使用时会引发异常。
createFromArray(int[][]) 方法的文档 似乎没有解释这一点。
英文:
I created a custom DataSetIterator. It works by randomly generating two INDArrays (one for input and one for output) in the next method and creating a DataSet out of it:
int[][] inputArray = new int[num][NUM_INPUTS];
int[][] expectedOutputArray = new int[num][];
for (int i = 0; i < num; i++) {//just fill the arrays with some data
int sum = 0;
int product = 1;
for (int j = 0; j < inputArray[i].length; j++) {
inputArray[i][j] = rand.nextInt();
sum += inputArray[i][j];
product *= inputArray[i][j];
}
expectedOutputArray[i] = new int[] { sum, product, sum / inputArray[i].length };
}
INDArray inputs = Nd4j.createFromArray(inputArray);//never closed
INDArray desiredOutputs = Nd4j.createFromArray(expectedOutputArray);//never closed
return new DataSet(inputs, desiredOutputs);
However, INDArray implements AutoClosable and the javadoc for close() states:
> This method releases exclusive off-heap resources uses by this INDArray instance. If INDArray relies on shared resources, exception will be thrown instead PLEASE NOTE: This method is NOT safe by any means
-
Do I need to close the
INDArrays? -
If so, when do I need to close the
INDArrays?
I have tried to use a try-with-resources but it threw an exception as the INDArray is closed when using it in the fit method.
The documentation of createFromArray(int[][]) does not seem to explain this.
答案1
得分: 2
你实际上不需要手动关闭它们。我们使用javacpp会自动处理这一过程。你可以选择手动关闭它们，但AutoCloseable接口的实现是为那些希望更多地控制ndarrays内存管理的人而设计的。
编辑：Javacpp是我们用于连接C++等本地库的底层本地集成技术。我们所有的计算和数据都基于本地代码和非托管内存。
调用close()方法只是强制我们更快地释放这些缓冲区。然而，Javacpp已经内置了自动释放缓冲区的功能。
英文:
you don't really need to close them. We take care of that automatically with javacpp. You can choose to close them but AutoCloseable was implemented for people who wanted more control over the memory management of the ndarrays.
Edit: Javacpp is the underlying native integration that we use to connect to native libraries we maintain written in c++ and other libraries. All of our calculations and data are all based on native code and off heap memory.
close() just forces us to de allocate those buffers faster. Javacpp has automatic de allocation built in to it already though.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论