共享库在使用R中的并行foreach时未被识别。

huangapple
go评论99阅读模式
英文:

Shared library is not recognized when using parallel foreach in R

问题

你在我的R代码中使用了一个用C语言编写的共享库。我使用dyn.load命令加载了编译后的共享库。我打算在一个并行的foreach循环中调用一个共享库函数。以下是我的代码：

  1. library(foreach)
  2. library(doParallel)
  4. totalCores = detectCores()
  5. cluster <- makeCluster(totalCores[1]-1)
  6. registerDoParallel(cluster)
  8. dyn.load("package.so")
  10. run <- function(i) {
  11.     row <- data[i,]
  12.     res <- .Call("c_function", as.double(row))
  13.     return(res)
  14. }
  16. result <- foreach(i=1:nrow(data), .combine = rbind) %dopar% {
  17.   run(i)
  18. }

我得到了以下错误：

  1. Error in { : 
  2.   task 1 failed - "C symbol name "c_function" not in load table"

尽管我已经加载了共享库，但似乎在并行任务中无法识别c_function。当然，如果我在foreach循环中让dyn.load命令，问题就解决了：

  1. result <- foreach(i=1:nrow(data), .combine = rbind) %dopar% {
  2.   dyn.load("package.so")
  3.   run(i)
  4. }

但我不确定这是否是最佳实践，因为在每次迭代中都会加载共享库（package.so），这可能效率不高。有什么想法吗？

编辑：

关于r2even的答案，我测试了以下代码：

  1. foreach(i=1:50,.packages='rootSolve') %dopar% {
  2.   print(is.loaded("c_function"))
  3. }

我的电脑有10个CPU核心（20个线程），所以在执行此代码时，totalCores变量的值是20。以下是结果：

  1. [[1]]
  2. [1] TRUE
  4. [[2]]
  5. [1] TRUE
  7. [[3]]
  8. [1] TRUE
  10. [[4]]
  11. [1] TRUE
  13. [[5]]
  14. [1] TRUE
  16. [[6]]
  17. [1] TRUE
  19. [[7]]
  20. [1] TRUE
  22. [[8]]
  23. [1] TRUE
  25. [[9]]
  26. [1] TRUE
  28. [[10]]
  29. [1] TRUE
  31. [[11]]
  32. [1] FALSE
  34. [[12]]
  35. [1] FALSE
  37. [[13]]
  38. [1] FALSE
  40. [[14]]
  41. [1] FALSE
  43. [[15]]
  44. [1] FALSE
  46. [[16]]
  47. [1] FALSE
  49. [[17]]
  50. [1] FALSE
  52. [[18]]
  53. [1] FALSE
  55. [[19]]
  56. [1] FALSE
  58. [[20]]
  59. [1] TRUE
  61. [[21]]
  62. [1] TRUE
  64. [[22]]
  65. [1] TRUE
  67. [[23]]
  68. [1] TRUE
  70. [[24]]
  71. [1] TRUE
  73. [[25]]
  74. [1] TRUE
  76. [[26]]
  77. [1] TRUE
  79. [[27]]
  80. [1] TRUE
  82. [[28]]
  83. [1] TRUE
  85. [[29]]
  86. [1] TRUE
  88. [[30]]
  89. [1] TRUE
  91. [[31]]
  92. [1] TRUE
  94. [[32]]
  95. [1] TRUE
  97. [[33]]
  98. [1] TRUE
  100. [[34]]
  101. [1] TRUE
  103. [[35]]
  104. [1] TRUE
  106. [[36]]
  107. [1] TRUE
  109. [[37]]
  110. [1] TRUE
  112. [[38]]
  113. [1] TRUE
  115. [[39]]
  116. [1] TRUE
  118. [[40]]
  119. [1] TRUE
  121. [[41]]
  122. [1] TRUE
  124. [[42]]
  125. [1] TRUE
  127. [[43]]
  128. [1] TRUE
  130. [[44]]
  131. [1] TRUE
  133. [[45]]
  134. [1] TRUE
  136. [[46]]
  137. [1] TRUE
  139. [[47]]
  140. [1] TRUE
  142. [[48]]
  143. [1] TRUE
  145. [[49]]
  146. [1] TRUE
  148. [[50]]
  149. [1] TRUE

这引发了一些问题。is.loaded("c_function")的值是否总是在迭代11到20之间为False？从第21次迭代开始是否保证它总是为True？

英文:

I am using a shared library written in C in my R code. I load the compiled shared library using dyn.load command. I am going to call a shared library function in a parallelized foreach loop. Here is my code:

  1. library(foreach)
  2. library(doParallel)
  4. totalCores = detectCores()
  5. cluster &lt;- makeCluster(totalCores[1]-1)
  6. registerDoParallel(cluster)
  8. dyn.load(&quot;package.so&quot;)
  10. run &lt;- function(i) {
  11.     row &lt;- data[i,]
  12.     res &lt;- .Call(&quot;c_function&quot;, as.double(row))
  13.     return(res)
  14. }
  17. result &lt;- foreach(i=1:nrow(data), .combine = rbind) %dopar% {
  18.   run(i)
  19. }

I get the following error:

  1. Error in { :
  2.   task 1 failed - &quot;C symbol name &quot;c_function&quot; not in load table&quot;

Although I have loaded the shared library, it seems c_function is not recognized in the parallel tasks. Of course when I let the dyn.load command in the foreach loop the problem is solved:

  1. result &lt;- foreach(i=1:nrow(data), .combine = rbind) %dopar% {
  2.   dyn.load(&quot;package.so&quot;)
  3.   run(i)
  4. }

But I am not sure if this is the best practice since at each iteration the shared library (package.so) is loaded and it may be not efficient. Any ideas?

Edit:

Regarding to r2even's answer I tested the following code:

  1. foreach(i=1:50,.packages=&#39;rootSolve&#39;) %dopar% {
  2.   print(is.loaded(&quot;c_function&quot;))
  3. }

My PC has 10 CPU cores (20 threads) so the value of totalCores variable was 20 when I executed this code. Here is the result:

  1. [[1]]
  2. [1] TRUE
  4. [[2]]
  5. [1] TRUE
  7. [[3]]
  8. [1] TRUE
  10. [[4]]
  11. [1] TRUE
  13. [[5]]
  14. [1] TRUE
  16. [[6]]
  17. [1] TRUE
  19. [[7]]
  20. [1] TRUE
  22. [[8]]
  23. [1] TRUE
  25. [[9]]
  26. [1] TRUE
  28. [[10]]
  29. [1] TRUE
  31. [[11]]
  32. [1] FALSE
  34. [[12]]
  35. [1] FALSE
  37. [[13]]
  38. [1] FALSE
  40. [[14]]
  41. [1] FALSE
  43. [[15]]
  44. [1] FALSE
  46. [[16]]
  47. [1] FALSE
  49. [[17]]
  50. [1] FALSE
  52. [[18]]
  53. [1] FALSE
  55. [[19]]
  56. [1] FALSE
  58. [[20]]
  59. [1] TRUE
  61. [[21]]
  62. [1] TRUE
  64. [[22]]
  65. [1] TRUE
  67. [[23]]
  68. [1] TRUE
  70. [[24]]
  71. [1] TRUE
  73. [[25]]
  74. [1] TRUE
  76. [[26]]
  77. [1] TRUE
  79. [[27]]
  80. [1] TRUE
  82. [[28]]
  83. [1] TRUE
  85. [[29]]
  86. [1] TRUE
  88. [[30]]
  89. [1] TRUE
  91. [[31]]
  92. [1] TRUE
  94. [[32]]
  95. [1] TRUE
  97. [[33]]
  98. [1] TRUE
  100. [[34]]
  101. [1] TRUE
  103. [[35]]
  104. [1] TRUE
  106. [[36]]
  107. [1] TRUE
  109. [[37]]
  110. [1] TRUE
  112. [[38]]
  113. [1] TRUE
  115. [[39]]
  116. [1] TRUE
  118. [[40]]
  119. [1] TRUE
  121. [[41]]
  122. [1] TRUE
  124. [[42]]
  125. [1] TRUE
  127. [[43]]
  128. [1] TRUE
  130. [[44]]
  131. [1] TRUE
  133. [[45]]
  134. [1] TRUE
  136. [[46]]
  137. [1] TRUE
  139. [[47]]
  140. [1] TRUE
  142. [[48]]
  143. [1] TRUE
  145. [[49]]
  146. [1] TRUE
  148. [[50]]
  149. [1] TRUE

It raises several questions. Is the value of is.loaded("c_function") always False only in the the iterations 11 to 20? Is it guaranteed that from the iteration 21 to the rest it is always true?

答案1

得分: 1

尝试这个hack，几乎肯定比重复调用dyn.load要高效得多：

  1. result <- foreach(i=1:nrow(data), .combine = rbind) %dopar% {
  2.   if (!is.loaded("c_function")) dyn.load("package.so")
  3.   run(i)
  4. }

（未经测试。）

英文:

Try this *hack*, almost certainly less inefficient than repeated calls to dyn.load:

  1. result &lt;- foreach(i=1:nrow(data), .combine = rbind) %dopar% {
  2.   if (!is.loaded(&quot;c_function&quot;)) dyn.load(&quot;package.so&quot;)
  3.   run(i)
  4. }

(Untested.)

huangapple
  • 本文由 发表于 2023年5月25日 16:58:32
  • 转载请务必保留本文链接：https://go.coder-hub.com/76330516.html
  • dll
  • parallel-foreach
  • parallel-processing
  • r
  • shared-libraries
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定