如何从管道内或管道运行之前获取数据流作业ID?

huangapple go评论74阅读模式
英文:

How do I get a dataflow Job ID from within the pipeline or before the pipeline ran?

问题

I'd like to grab the job id here

在这里我想获取作业标识符

英文:

I want to grab the dataflow's Job ID as a string before actually running the pipeline or inside of it.

As an example let's say I have this:

DataflowOptions options = optionsConst.getOptions();         
Pipeline p = Pipeline.create(options);
String jobID = doSomthing(); <-- I'd like to grab the job id here

PCollection<KV<String, Object>> refData = p.apply("BigTable Access", BigtableIO.read()
                .withProjectId(options.getBigtableProjectId())
                .withInstanceId(options.getBigtableInstanceId())
                .withTableId(options.getLookupTableId()))
                .apply("Keep tags valid versions", ParDo.of(new MethodThatDoesSomethingWithJobID(**jobId**))); <-- Or inside this

答案1

得分: 1

不可能做到这一点,因为作业 ID 是在您开始运行作业时创建的。您可以在开始运行管道后保存输出,或返回正在运行的作业列表。

英文:

Is not possible to do that, as the Job Id is created once you start to run it. What you can do is to save the output once you start run the pipeline, or to return the list job running

huangapple
  • 本文由 发表于 2020年8月14日 16:52:32
  • 转载请务必保留本文链接:https://go.coder-hub.com/63409564.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定