如何在应用程序模式下将 Java 参数传递给 Flink 作业构件

huangapple go评论75阅读模式
英文:

How to pass java arguments to Flink job artifacts in Application Mode

问题

我刚刚将Flink从版本1.10升级到1.11。在1.11中,Flink提供了新功能,用户可以在Kubernetes上以应用程序模式部署作业。
链接:https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/deployment/kubernetes.html#deploy-session-cluster

在V1.10中,我们启动Flink K8s集群,然后通过以下命令将作业提交给Flink:

exec ./bin/flink run \
  -d \
  /streakerflink_deploy.jar \
    --arg1 blablabla
    --arg2 blablabla
    --arg3 blablabla
    ...

我们通过这个命令传递Java参数。

但是,在V1.11中,如果我们运行应用程序模式,就不需要运行上述的flink run命令。我想知道在应用程序模式(也称为作业集群)下,我们如何向Flink作业传递参数?

任何帮助将不胜感激!

英文:

I just upgrade the Flink from version 1.10 to 1.11. In 1.11, Flink provides new features that users can deploy the job in Application Mode on Kubernetes.
https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/deployment/kubernetes.html#deploy-session-cluster

In V1.10, we start the Flink K8s cluster and then submit the job to Flink by run

exec ./bin/flink run \
  -d \
  /streakerflink_deploy.jar \
    --arg1 blablabla
    --arg2 blablabla
    --arg3 blablabla
    ...

We pass the java arguments through this command.

But, in V1.11, if we run Application mode, we don't need to run the flink run command above. I am wondering how do we pass the arguments to Flink job in Application mode (aka Job Cluster)?

Any help will be appreciated!

答案1

得分: 1

Flink在Kubernetes上的应用模式在文档中有描述。您需要创建一个包含作业的Docker镜像。如文档所述,可以使用./bin/flink run-application [...]来执行作业。

英文:

Flink's application mode on Kubernetes is described in the docs. You have to create a Docker image containing your job. The job can be executed using ./bin/flink run-application [...] as described in the docs.

答案2

得分: 1

因为您正在使用Helm Chart在Kubernetes(也称为K8s)上启动Flink集群,我假设您在谈论独立的K8s模式。实际上,应用程序模式与1.10版本及之前的作业集群非常相似。
因此,您可以在jobmanager-job.yaml中的args字段中设置作业参数,就像下面这样。

...
args: ["standalone-job", "--job-classname", "org.apache.flink.streaming.examples.join.WindowJoin", "--windowSize", "3000", "--rate", "100"]
...

如果您真的指的是原生K8s模式,则可以在flink run-application命令之后直接添加。

$ ./bin/flink run-application -p 8 -t kubernetes-application \
  -Dkubernetes.cluster-id=<ClusterId> \
  -Dtaskmanager.memory.process.size=4096m \
  -Dkubernetes.taskmanager.cpu=2 \
  -Dtaskmanager.numberOfTaskSlots=4 \
  -Dkubernetes.container.image=<CustomImageName> \
  local:///opt/flink/examples/streaming/WindowJoin.jar \
  --windowSize 3000 --rate 100

注意:
请记住独立K8s模式与原生K8s模式之间的主要区别在于动态资源分配。在原生模式中,我们有一个内嵌的K8s客户端,因此Flink JobManager可以根据需求分配/释放TaskManager pod。目前,原生模式只能在Flink命令(kubernetes-session.shflink run-application)中使用。

英文:

Since you are using helm chart to start the Flink cluster on Kubernetes(aka K8s), i assume you are talking about the standalone K8s mode. Actually, the Application mode is very similar to the job cluster in 1.10 and before.
So you could set the job arguments in the args field in the jobmanager-job.yaml just like following.

...
args: [&quot;standalone-job&quot;, &quot;--job-classname&quot;, &quot;org.apache.flink.streaming.examples.join.WindowJoin&quot;, &quot;--windowSize&quot;, &quot;3000&quot;, &quot;--rate&quot;, &quot;100&quot;]
...

If you really mean the native K8s mode, then it could be directly added after the flink run-application command.

$ ./bin/flink run-application -p 8 -t kubernetes-application \
  -Dkubernetes.cluster-id=&lt;ClusterId&gt; \
  -Dtaskmanager.memory.process.size=4096m \
  -Dkubernetes.taskmanager.cpu=2 \
  -Dtaskmanager.numberOfTaskSlots=4 \
  -Dkubernetes.container.image=&lt;CustomImageName&gt; \
  local:///opt/flink/examples/streaming/WindowJoin.jar \
  --windowSize 3000 --rate 100

Note:
Please keep in mind that the key difference between standalone K8s and native K8s mode is the dynamic resource allocation. In native mode, we have an embedded K8s client, so Flink JobManager could allocate/release TaskManager pods on demands. Currently, native mode could only be used in Flink commands(kubernetes-session.sh, flink run-application).

huangapple
  • 本文由 发表于 2020年9月2日 15:25:42
  • 转载请务必保留本文链接:https://go.coder-hub.com/63700645.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定