无法在Spring Boot中创建SparkSession。

huangapple
go评论89阅读模式
英文:

Can't create SparkSession in Spring Boot

问题

我正在尝试在Spring上运行Spark，项目本身没有错误，但在尝试创建会话后，我遇到了这个错误。

  1. java.lang.NoSuchMethodError: 'scala.collection.SeqOps scala.collection.mutable.Buffer$.empty()'

pom.xml

  1. <dependencies>
  2.     <dependency>
  3.         <groupId>org.springframework.boot</groupId>
  4.         <artifactId>spring-boot-starter-web</artifactId>
  5.     </dependency>
  6.     <dependency>
  7.         <groupId>com.thoughtworks.paranamer</groupId>
  8.         <artifactId>paranamer</artifactId>
  9.         <version>2.8</version>
  10.     </dependency>
  11.     <!-- https://mvnrepository.com/artifact/org.springframework/spring-core -->
  12.     <dependency>
  13.         <groupId>org.springframework</groupId>
  14.         <artifactId>spring-core</artifactId>
  15.         <version>6.0.5</version>
  16.     </dependency>
  17.     <dependency>
  18.         <groupId>org.apache.spark</groupId>
  19.         <artifactId>spark-core_2.12</artifactId>
  20.         <version>3.3.0</version>
  21.         <exclusions>
  22.             <exclusion>
  23.                 <groupId>org.slf4j</groupId>
  24.                 <artifactId>slf4j-log4j12</artifactId>
  25.             </exclusion>
  26.             <exclusion>
  27.                 <groupId>log4j</groupId>
  28.                 <artifactId>log4j</artifactId>
  29.             </exclusion>
  30.         </exclusions>
  31.     </dependency>
  32.     <dependency>
  33.         <groupId>org.springframework.boot</groupId>
  34.         <artifactId>spring-boot-starter-test</artifactId>
  35.         <scope>test</scope>
  36.     </dependency>
  37.     <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-sql -->
  38.     <dependency>
  39.         <groupId>org.apache.spark</groupId>
  40.         <artifactId>spark-sql_2.13</artifactId>
  41.         <version>3.3.2</version>
  42.         <scope>provided</scope>
  43.     </dependency>
  44.     <dependency>
  45.         <groupId>org.scala-lang</groupId>
  46.         <artifactId>scala-library</artifactId>
  47.         <version>2.12.15</version>
  48.     </dependency>
  49. </dependencies>

controller

  1. @RestController
  2. public class MainController {
  3.     @GetMapping("/")
  4.     public String index() {
  5.         SparkSession spark = SparkSession
  6.                 .builder()
  7.                 .appName("AppSpark")
  8.                 .config("spark.master", "local")
  9.                 .getOrCreate();
  11.         return "Hello World!";
  12.     }
  13. }

我尝试降低pom文件中的版本，但没有帮助，只是错误类型发生了变化。如果有人遇到相同的错误，请帮忙解决将不胜感激。

英文:

I'm trying to run Spark on Spring, the project itself runs without errors, but after I try to create a session, i get this error.

  1. java.lang.NoSuchMethodError: &#39;scala.collection.SeqOps scala.collection.mutable.Buffer$.empty()&#39;

pom.xml

  1.     &lt;dependencies&gt;
  3.             &lt;dependency&gt;
  4.                 &lt;groupId&gt;org.springframework.boot&lt;/groupId&gt;
  5.                 &lt;artifactId&gt;spring-boot-starter-web&lt;/artifactId&gt;
  6.             &lt;/dependency&gt;
  7.             &lt;dependency&gt;
  8.                 &lt;groupId&gt;com.thoughtworks.paranamer&lt;/groupId&gt;
  9.                 &lt;artifactId&gt;paranamer&lt;/artifactId&gt;
  10.                 &lt;version&gt;2.8&lt;/version&gt;
  11.             &lt;/dependency&gt;
  12.         &lt;!-- https://mvnrepository.com/artifact/org.springframework/spring-core --&gt;
  13.         &lt;dependency&gt;
  14.             &lt;groupId&gt;org.springframework&lt;/groupId&gt;
  15.             &lt;artifactId&gt;spring-core&lt;/artifactId&gt;
  16.             &lt;version&gt;6.0.5&lt;/version&gt;
  17.         &lt;/dependency&gt;
  19.         &lt;dependency&gt;
  20.             &lt;groupId&gt;org.apache.spark&lt;/groupId&gt;
  21.             &lt;artifactId&gt;spark-core_2.12&lt;/artifactId&gt;
  22.             &lt;version&gt;3.3.0&lt;/version&gt;
  23.             &lt;exclusions&gt;
  24.                 &lt;exclusion&gt;
  25.                     &lt;groupId&gt;org.slf4j&lt;/groupId&gt;
  26.                     &lt;artifactId&gt;slf4j-log4j12&lt;/artifactId&gt;
  27.                 &lt;/exclusion&gt;
  28.                 &lt;exclusion&gt;
  29.                     &lt;groupId&gt;log4j&lt;/groupId&gt;
  30.                     &lt;artifactId&gt;log4j&lt;/artifactId&gt;
  31.                 &lt;/exclusion&gt;
  33.             &lt;/exclusions&gt;
  34.         &lt;/dependency&gt;
  36.         &lt;dependency&gt;
  37.                 &lt;groupId&gt;org.springframework.boot&lt;/groupId&gt;
  38.                 &lt;artifactId&gt;spring-boot-starter-test&lt;/artifactId&gt;
  39.                 &lt;scope&gt;test&lt;/scope&gt;
  40.             &lt;/dependency&gt;
  41.         &lt;!-- https://mvnrepository.com/artifact/org.apache.spark/spark-sql --&gt;
  42.         &lt;dependency&gt;
  43.             &lt;groupId&gt;org.apache.spark&lt;/groupId&gt;
  44.             &lt;artifactId&gt;spark-sql_2.13&lt;/artifactId&gt;
  45.             &lt;version&gt;3.3.2&lt;/version&gt;
  46.             &lt;scope&gt;provided&lt;/scope&gt;
  47.         &lt;/dependency&gt;
  49.         &lt;dependency&gt;
  50.             &lt;groupId&gt;org.scala-lang&lt;/groupId&gt;
  51.             &lt;artifactId&gt;scala-library&lt;/artifactId&gt;
  52.             &lt;version&gt;2.12.15&lt;/version&gt;
  53.         &lt;/dependency&gt;
  54.     &lt;/dependencies&gt;

controller

  1. @RestController
  2. public class MainController {
  3.     @GetMapping(&quot;/&quot;)
  4.     public String index() {
  5.         SparkSession spark = SparkSession
  6.                 .builder()
  7.                 .appName(&quot;AppSpark&quot;)
  8.                 .config(&quot;spark.master&quot;, &quot;local&quot;)
  9.                 .getOrCreate();
  11.         return &quot;Hello World!&quot;;
  12.     }
  13. }

I tried to downgrade the versions in the pom file, but it did not help, only the error type changed.

If anyone has come across the same error, I would appreciate any help.

答案1

得分: 1

Scala库是针对特定Scala版本编译的，这个版本在构件名称中可见，比如spark-sql_2.13表示它是为Scala 2.13编译的spark-sql。

在同一个项目中不能使用多个Scala版本。

在你的情况下，你有：

  • scala-library: 2.12
  • spark-core: 2.12
  • spark-sql: 2.13

选择一个版本并坚持使用它，如果可能的话，选择2.13，因为它更近期。

英文:

Scala libraries are compiled against a Scala version which is visible in the artifact name like spark-sql_2.13 means it's spark-sql compiled for Scala 2.13.

You cannot have multiple Scala versions in the same project.

In your case, you have:

  • scala-library: 2.12
  • spark-core: 2.12
  • spark.sql: 2.13

Choose a single version and stick to it. 2.13 if possible as it's more recent.

huangapple
  • 本文由 发表于 2023年3月3日 23:32:41
  • 转载请务必保留本文链接：https://go.coder-hub.com/75629038.html
  • apache-spark
  • java
  • scala
  • spring
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定