我如何在BEAM工作节点上使用Log4j2日志记录?

huangapple go评论75阅读模式
英文:

How can I use Log4j2 logging on BEAM workers?

问题

I'm trying to use Log4j2 logging implementation together with Slf4j api in the BEAM pipeline. In maven it looks like this:

    <dependency>
      <groupId>org.apache.logging.log4j</groupId>
      <artifactId>log4j-core</artifactId>
      <version>2.20.0</version>
    </dependency>
    <dependency>
      <groupId>org.apache.logging.log4j</groupId>
      <artifactId>log4j-slf4j2-impl</artifactId>
      <version>2.20.0</version>
    </dependency>

It works fine locally. Also the logging works fine for the container where I start the pipeline. However, inside the workers this setup causes some issues:

Caused by: org.apache.logging.log4j.LoggingException: log4j-slf4j2-impl cannot be present with log4j-to-slf4j

This additional library log4j-to-slf4j is not part of my project and it's not listed in dependencies (also not in transitive dependencies). Looks like it's added by BEAM sdk itself. The question is: how can I configure BEAM to use my version of Log4j?

英文:

I'm trying to use Log4j2 logging implementation together with Slf4j api in the BEAM pipeline. In maven it looks like this:

    <dependency>
      <groupId>org.apache.logging.log4j</groupId>
      <artifactId>log4j-core</artifactId>
      <version>2.20.0</version>
    </dependency>
    <dependency>
      <groupId>org.apache.logging.log4j</groupId>
      <artifactId>log4j-slf4j2-impl</artifactId>
      <version>2.20.0</version>
    </dependency>

It works fine locally. Also the logging works fine for the container where I start the pipeline. However, inside the workers this setup causes some issues:

Caused by: org.apache.logging.log4j.LoggingException: log4j-slf4j2-impl cannot be present with log4j-to-slf4j

This additional library log4j-to-slf4j is not part of my project and it's not listed in dependencies (also not in transitive dependencies). Looks like it's added by BEAM sdk itself. The question is: how can I configure BEAM to use my version of Log4j?

答案1

得分: 0

当我们使用 Maven原型 生成 2.47.0 版本的 Beam 项目时:

mvn archetype:generate \
    -DarchetypeGroupId=org.apache.beam \
    -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
    -DarchetypeVersion=2.47.0 \
    -DgroupId=org.example \
    -DartifactId=word-count-beam \
    -Dversion="0.1" \
    -Dpackage=org.apache.beam.examples \
    -DinteractiveMode=false

它将在 pom.xml 文件中生成以下依赖关系:

<properties>
      <beam.version>2.47.0</beam.version>
      <slf4j.version>1.7.30</slf4j.version>
</properties>

<dependency>
       <groupId>org.slf4j</groupId>
       <artifactId>slf4j-api</artifactId>
       <version>${slf4j.version}</version>
</dependency>

<dependency>
       <groupId>org.slf4j</groupId>
       <artifactId>slf4j-jdk14</artifactId>
       <version>${slf4j.version}</version>
       <!-- When loaded at runtime this will wire up slf4j to the JUL backend -->
       <scope>runtime</scope>
</dependency>

然后在代码中,你可以像在Java项目中一样使用 SL4J

public class TeamLeagueApp {
    private static final Logger LOGGER = LoggerFactory.getLogger(TeamLeagueApp.class);

    ....
    LOGGER.info("My log");
英文:

When we generate a Beam project with 2.47.0 version using the Maven archetype :

mvn archetype:generate \
    -DarchetypeGroupId=org.apache.beam \
    -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
    -DarchetypeVersion=2.47.0 \
    -DgroupId=org.example \
    -DartifactId=word-count-beam \
    -Dversion=&quot;0.1&quot; \
    -Dpackage=org.apache.beam.examples \
    -DinteractiveMode=false

It generate the project with the following dependencies in the pom.xml file :

&lt;properties&gt;
      &lt;beam.version&gt;2.47.0&lt;/beam.version&gt;
      &lt;slf4j.version&gt;1.7.30&lt;/slf4j.version&gt;
&lt;/properties&gt;

&lt;dependency&gt;
       &lt;groupId&gt;org.slf4j&lt;/groupId&gt;
       &lt;artifactId&gt;slf4j-api&lt;/artifactId&gt;
       &lt;version&gt;${slf4j.version}&lt;/version&gt;
&lt;/dependency&gt;

&lt;dependency&gt;
       &lt;groupId&gt;org.slf4j&lt;/groupId&gt;
       &lt;artifactId&gt;slf4j-jdk14&lt;/artifactId&gt;
       &lt;version&gt;${slf4j.version}&lt;/version&gt;
       &lt;!-- When loaded at runtime this will wire up slf4j to the JUL backend --&gt;
       &lt;scope&gt;runtime&lt;/scope&gt;
&lt;/dependency&gt;

Then in the code, you use SL4J as usual in Java projects :

public class TeamLeagueApp {
    private static final Logger LOGGER = LoggerFactory.getLogger(TeamLeagueApp.class);

    ....
    LOGGER.info(&quot;My log&quot;);

I recommend you using the dependencies and version brought by Beam natively to handle logs, in order to prevent some dependencies conflicts and issues.

答案2

得分: 0

所以解决方案很简单。看起来BEAM使用slf4j v1库,所以这些依赖项与之配合良好,并使用log4j2作为日志的后端:

    &lt;dependency&gt;
      &lt;groupId&gt;org.apache.logging.log4j&lt;/groupId&gt;
      &lt;artifactId&gt;log4j-core&lt;/artifactId&gt;
      &lt;version&gt;${log4j.version}&lt;/version&gt;
    &lt;/dependency&gt;
    &lt;dependency&gt;
      &lt;groupId&gt;org.apache.logging.log4j&lt;/groupId&gt;
      &lt;artifactId&gt;log4j-slf4j-impl&lt;/artifactId&gt;
      &lt;version&gt;${log4j.version}&lt;/version&gt;
    &lt;/dependency&gt;
英文:

So the solution is easy. Looks like BEAM uses slf4j v1 libs so these dependencies work fine with it and use log4j2 as a backend for logging:

    &lt;dependency&gt;
      &lt;groupId&gt;org.apache.logging.log4j&lt;/groupId&gt;
      &lt;artifactId&gt;log4j-core&lt;/artifactId&gt;
      &lt;version&gt;${log4j.version}&lt;/version&gt;
    &lt;/dependency&gt;
    &lt;dependency&gt;
      &lt;groupId&gt;org.apache.logging.log4j&lt;/groupId&gt;
      &lt;artifactId&gt;log4j-slf4j-impl&lt;/artifactId&gt;
      &lt;version&gt;${log4j.version}&lt;/version&gt;
    &lt;/dependency&gt;

huangapple
  • 本文由 发表于 2023年5月10日 17:58:44
  • 转载请务必保留本文链接:https://go.coder-hub.com/76217094.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定