英文:
Could not transfer artifact io.confluent:kafka-connect-storage-common-parent:pom:6.0.0-SNAPSHOT from/to confluent (${confluent.maven.repo})
问题
以下是翻译好的部分:
我第一次尝试使用Kafka Connect,并且我想要将SAP S/4 HANA连接到Hive。我已经使用以下链接创建了SAP S/4源Kafka连接器:
https://github.com/SAP/kafka-connect-sap
但是,我无法创建一个HDFS汇聚连接器。问题与pom文件有关。
我已经尝试了mvn clean package
命令。但是,我遇到了以下错误:
[INFO] 正在扫描项目...
[ERROR] [ERROR] 处理POM文件时遇到一些问题:
[FATAL] 无法解析 io.confluent:kafka-connect-hdfs:[unknown-version] 的父POM。无法从 confluent (${confluent.maven.repo}) 传输 artifact io.confluent:kafka-connect-storage-common-parent:pom:6.0.0-SNAPSHOT,因为可用的连接器工厂无法访问 ${confluent.maven.repo} 以使用默认类型:BasicRepositoryConnectorFactory,而 'parent.relativePath' 指向错误的本地POM,位于第 22 行,第 13 列
@
[ERROR] 生成的项目 io.confluent:kafka-connect-hdfs:[unknown-version] (/home/dev_dice_sa/test-dicestreaming/config/kafka-connect-hdfs/pom.xml) 遇到 1 个错误
[ERROR] 无法解析 io.confluent:kafka-connect-hdfs:[unknown-version] 的父 POM。无法从 confluent (${confluent.maven.repo}) 传输 artifact io.confluent:kafka-connect-storage-common-parent:pom:6.0.0-SNAPSHOT,因为可用的连接器工厂无法访问 ${confluent.maven.repo} 以使用默认类型:BasicRepositoryConnectorFactory,位于第 22 行,第 13 列:无法使用已注册的 transporter 工厂访问 ${confluent.maven.repo}:WagonTransporterFactory:不支持的传输协议 -> [帮助 2]
[ERROR]
[ERROR] 要查看错误的完整堆栈跟踪,请使用 -e 开关重新运行 Maven。
[ERROR] 使用 -X 开关重新运行 Maven 以启用完整的调试日志记录。
[ERROR]
[ERROR] 有关错误和可能的解决方法的更多信息,请阅读以下文章:
[ERROR] [帮助 1] http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
[ERROR] [帮助 2] http://cwiki.apache.org/confluence/display/MAVEN/UnresolvableModelException
这是我的pom.xml文件内容:
<?xml version="1.0" encoding="UTF-8"?>
<!-- ... 以下省略 ... -->
我已经查阅了以下链接:
https://github.com/confluentinc/kafka-connect-hdfs/wiki/FAQ
但是,我无法理解其中的内容。请帮助我解决这个问题。
英文:
I am trying Kafka connect for the first time and I want to connect SAP S/4 HANA to Hive. I have created the SAP S/4 source Kafka connector using this:
https://github.com/SAP/kafka-connect-sap
But, I am not able to create an HDFS sink connector. The issue is related to pom file.
I have tried mvn clean package
.
But, I got this error:
[INFO] Scanning for projects...
[ERROR] [ERROR] Some problems were encountered while processing the POMs:
[FATAL] Non-resolvable parent POM for io.confluent:kafka-connect-hdfs:[unknown-version]: Could not transfer artifact io.confluent:kafka-connect-storage-common-parent:pom:6.0.0-SNAPSHOT from/to confluent (${confluent.maven.repo}): Cannot access ${confluent.maven.repo} with type default using the available connector factories: BasicRepositoryConnectorFactory and 'parent.relativePath' points at wrong local POM @ line 22, column 13
@
[ERROR] The build could not read 1 project -> [Help 1]
[ERROR]
[ERROR] The project io.confluent:kafka-connect-hdfs:[unknown-version] (/home/dev_dice_sa/test-dicestreaming/config/kafka-connect-hdfs/pom.xml) has 1 error
[ERROR] Non-resolvable parent POM for io.confluent:kafka-connect-hdfs:[unknown-version]: Could not transfer artifact io.confluent:kafka-connect-storage-common-parent:pom:6.0.0-SNAPSHOT from/to confluent (${confluent.maven.repo}): Cannot access ${confluent.maven.repo} with type default using the available connector factories: BasicRepositoryConnectorFactory and 'parent.relativePath' points at wrong local POM @ line 22, column 13: Cannot access ${confluent.maven.repo} using the registered transporter factories: WagonTransporterFactory: Unsupported transport protocol -> [Help 2]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
[ERROR] [Help 2] http://cwiki.apache.org/confluence/display/MAVEN/UnresolvableModelException
This is my pom.xml:
<?xml version="1.0" encoding="UTF-8"?>
<!--
~ Copyright 2018 Confluent Inc.
~
~ Licensed under the Confluent Community License (the "License"); you may not use
~ this file except in compliance with the License. You may obtain a copy of the
~ License at
~
~ http://www.confluent.io/confluent-community-license
~
~ Unless required by applicable law or agreed to in writing, software
~ distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
~ WARRANTIES OF ANY KIND, either express or implied. See the License for the
~ specific language governing permissions and limitations under the License.
-->
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>io.confluent</groupId>
<artifactId>kafka-connect-storage-common-parent</artifactId>
<version>6.0.0-SNAPSHOT</version>
</parent>
<artifactId>kafka-connect-hdfs</artifactId>
<packaging>jar</packaging>
<name>kafka-connect-hdfs</name>
<organization>
<name>Confluent, Inc.</name>
<url>http://confluent.io</url>
</organization>
<url>http://confluent.io</url>
<description>
A Kafka Connect HDFS connector for copying data between Kafka and Hadoop HDFS.
</description>
<licenses>
<license>
<name>Confluent Community License</name>
<url>http://www.confluent.io/confluent-community-license</url>
<distribution>repo</distribution>
</license>
</licenses>
<scm>
<connection>scm:git:git://github.com/confluentinc/kafka-connect-hdfs.git</connection>
<developerConnection>scm:git:git@github.com:confluentinc/kafka-connect-hdfs.git</developerConnection>
<url>https://github.com/confluentinc/kafka-connect-hdfs</url>
<tag>HEAD</tag>
</scm>
<properties>
<confluent.maven.repo>http://packages.confluent.io/maven/</confluent.maven.repo>
<apacheds-jdbm1.version>2.0.0-M2</apacheds-jdbm1.version>
<kafka.connect.maven.plugin.version>0.11.1</kafka.connect.maven.plugin.version>
</properties>
<repositories>
<repository>
<id>confluent</id>
<name>Confluent</name>
<url>${confluent.maven.repo}</url>
</repository>
</repositories>
<dependencies>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>connect-api</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>connect-json</artifactId>
<version>${kafka.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>io.confluent</groupId>
<artifactId>kafka-connect-storage-common</artifactId>
<version>${confluent.version}</version>
</dependency>
<dependency>
<groupId>io.confluent</groupId>
<artifactId>kafka-connect-storage-core</artifactId>
<version>${confluent.version}</version>
</dependency>
<dependency>
<groupId>io.confluent</groupId>
<artifactId>kafka-connect-storage-format</artifactId>
<version>${confluent.version}</version>
</dependency>
<dependency>
<groupId>io.confluent</groupId>
<artifactId>kafka-connect-storage-partitioner</artifactId>
<version>${confluent.version}</version>
</dependency>
<dependency>
<groupId>io.confluent</groupId>
<artifactId>kafka-connect-storage-wal</artifactId>
<version>${confluent.version}</version>
</dependency>
<dependency>
<groupId>io.confluent</groupId>
<artifactId>kafka-connect-storage-hive</artifactId>
<version>${confluent.version}</version>
<exclusions>
<exclusion>
<groupId>io.netty</groupId>
<artifactId>netty</artifactId>
</exclusion>
<exclusion>
<groupId>io.netty</groupId>
<artifactId>netty-all</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>com.github.spotbugs</groupId>
<artifactId>spotbugs-annotations</artifactId>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-minicluster</artifactId>
<version>${hadoop.version}</version>
<scope>test</scope>
<exclusions>
<exclusion>
<groupId>io.netty</groupId>
<artifactId>netty</artifactId>
</exclusion>
<exclusion>
<groupId>io.netty</groupId>
<artifactId>netty-all</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-minikdc</artifactId>
<version>${hadoop.version}</version>
<scope>test</scope>
<exclusions>
<exclusion>
<!-- exclude this from hadoop minikdc as the minikdc depends on
the apacheds-jdbm1 bundle, which is not available in maven central-->
<groupId>org.apache.directory.jdbm</groupId>
<artifactId>apacheds-jdbm1</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<!-- add this to satisfy the dependency requirement of apacheds-jdbm1-->
<groupId>org.apache.directory.jdbm</groupId>
<artifactId>apacheds-jdbm1</artifactId>
<version>${apacheds-jdbm1.version}</version>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>io.confluent</groupId>
<version>${kafka.connect.maven.plugin.version}</version>
<artifactId>kafka-connect-maven-plugin</artifactId>
<executions>
<execution>
<goals>
<goal>kafka-connect</goal>
</goals>
<configuration>
<title>Kafka Connect HDFS</title>
<documentationUrl>https://docs.confluent.io/${project.version}/connect/connect-hdfs/docs/index.html</documentationUrl>
<description>
The HDFS connector allows you to export data from Kafka topics to HDFS files in a variety of formats and integrates with Hive to make data immediately available for querying with HiveQL.
The connector periodically polls data from Kafka and writes them to HDFS. The data from each Kafka topic is partitioned by the provided partitioner and divided into chunks. Each chunk of data is represented as an HDFS file with topic, Kafka partition, start and end offsets of this data chunk in the filename. If no partitioner is specified in the configuration, the default partitioner which preserves the Kafka partitioning is used. The size of each data chunk is determined by the number of records written to HDFS, the time written to HDFS and schema compatibility.
The HDFS connector integrates with Hive and when it is enabled, the connector automatically creates an external Hive partitioned table for each Kafka topic and updates the table according to the available data in HDFS.
</description>
<supportProviderName>Confluent, Inc.</supportProviderName>
<supportSummary>Confluent supports the HDFS sink connector alongside community members as part of its Confluent Platform offering.</supportSummary>
<supportUrl>https://docs.confluent.io/current/</supportUrl>
<supportLogo>logos/confluent.png</supportLogo>
<ownerUsername>confluentinc</ownerUsername>
<ownerType>organization</ownerType>
<ownerName>Confluent, Inc.</ownerName>
<ownerUrl>https://confluent.io/</ownerUrl>
<ownerLogo>logos/confluent.png</ownerLogo>
<dockerNamespace>confluentinc</dockerNamespace>
<dockerName>cp-kafka-connect</dockerName>
<dockerTag>${project.version}</dockerTag>
<componentTypes>
<componentType>sink</componentType>
</componentTypes>
<tags>
<tag>hadoop</tag>
<tag>hdfs</tag>
<tag>hive</tag>
</tags>
<confluentControlCenterIntegration>true</confluentControlCenterIntegration>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<compilerArgs>
<arg>-Xlint:all,-deprecation,-processing</arg>
<arg>-Werror</arg>
</compilerArgs>
<showWarnings>true</showWarnings>
<showDeprecation>false</showDeprecation>
</configuration>
</plugin>
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<configuration>
<descriptors>
<descriptor>src/assembly/development.xml</descriptor>
<descriptor>src/assembly/package.xml</descriptor>
</descriptors>
<attach>false</attach>
</configuration>
<executions>
<execution>
<id>make-assembly</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<reuseForks>false</reuseForks>
<forkCount>1</forkCount>
<systemPropertyVariables>
<datanucleus.schema.autoCreateAll>true</datanucleus.schema.autoCreateAll>
</systemPropertyVariables>
<additionalClasspathElements>
<!--puts hive-site.xml on classpath - w/o HMS tables are not created-->
<additionalClasspathElement>src/test/resources/conf</additionalClasspathElement>
</additionalClasspathElements>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-checkstyle-plugin</artifactId>
<executions>
<execution>
<id>validate</id>
<phase>validate</phase>
<configuration>
<suppressionsLocation>checkstyle/suppressions.xml</suppressionsLocation>
</configuration>
<goals>
<goal>check</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<artifactId>maven-clean-plugin</artifactId>
<version>3.0.0</version>
<configuration>
<filesets>
<fileset>
<directory>.</directory>
<includes>
<include>derby.log</include>
<include>metastore_db/</include>
</includes>
</fileset>
</filesets>
</configuration>
</plugin>
</plugins>
<resources>
<resource>
<directory>src/main/resources</directory>
<filtering>true</filtering>
</resource>
</resources>
</build>
<profiles>
<profile>
<id>standalone</id>
<build>
<plugins>
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<configuration>
<descriptors>
<descriptor>src/assembly/standalone.xml</descriptor>
</descriptors>
</configuration>
</plugin>
</plugins>
</build>
</profile>
</profiles>
</project>
I have looked into this link:
https://github.com/confluentinc/kafka-connect-hdfs/wiki/FAQ
But, I was not able to follow this. Please help me out.
答案1
得分: 1
我建议您下载现有的Confluent平台,其中已经包括了HDFS Connect。
否则,不要仅仅检出主分支来构建项目,而是检出一个发布版本。
英文:
I suggest you download existing Confluent Platform which includes HDFS Connect already
Otherwise, checkout a release version rather than only the master branch to build the project.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论