英文:
Ignite Community Edition: Configuring IGFS with HDFS as persistent storage
问题
I would like to have more details about the configuration of Apache Ignite (cluster IGFS) and HDFS. I can't find any official reference, so I doubt that it is possible to do so with the open-source version of Apache Ignite, and I need to switch to something like GridGain. Is that true? I would like to use Apache Ignite to perform in-memory computation with Spark, and I would like to have a "kind of automatic" sync with Hadoop HDFS as backend storage because I don't want to perform any manual load from HDFS.
Thanks.
英文:
i would like to have more details about the configuration of apache ignite (cluster igfs) and hdfs. I can't find any official reference, so i doubt that is possibile to do so with the opensource version of apache ignite and i need to switch to something like gridgain. Is that true? I would like to use apache ignite to perform in-memory computation with spark and i would like to have a "kind of automatic" sync with hadoop hdfs as backend storage, because i don't want to perform any manual load from hdfs.
thanks
答案1
得分: 4
你仍然可以使用Apache Ignite与Spark集成来处理HDFS:
目前有适用于Spark 2.3、2.4和3.0的集成。最近添加了后者,但出于某种原因它没有在文档中。但你可以在这里找到:
无论如何,你也可以查看我关于这个集成的网络研讨会:
此外,你可以在这里查看示例:
英文:
You can still use Apache Ignite's integration with Spark to work with HDFS:
There are currently integrations for Spark 2.3, 2.4 and 3.0. The latter was added not so long ago, for some reason it is not in the documentation. But it's here:
https://downloads.apache.org/ignite/ignite-extensions/ignite-spark-ext/3.0.0/
Anyway, you can also check my webinar about this integration:
https://www.youtube.com/watch?v=lkRh2TO8VSU
Also you can see the examples here:
答案2
得分: 1
Igfs在2019年被移除,以便专注于高性能的内存计算和查询。Apache HDFS项目更适合作为文件系统(en.wikipedia.org/wiki/Apache_Hadoop#HDFS)。
英文:
Igfs was removed in 2019 to focus on high-performance in-memory computing and querying. The Apache HDFS project is better suited as a file system (en.wikipedia.org/wiki/Apache_Hadoop#HDFS) .
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论