英文:
Error creating dataflow pipeline using custom template on GCP
问题
1 在选择模板路径后，我看到了以下警告或可能是错误信息：
模板元数据错误详细信息
无法作为 Flex 模板和传统模板处理。 Flex 模板处理结果：未提供 SDK 语言信息。传统模板处理结果：(35197769719996987970000d1381234f)：无法打开模板文件：gs://abc/extraction/templates/cron-template_metadata。
我确实有 gs://abc/extraction/templates/cron-template，因为这是我在参数中指定的 --templateLocation=gs://abc/extraction/templates/cron-template 文件，但没有 gs://abc/extraction/templates/cron-template_metadata。
我不确定这是警告还是错误。
2 在我点击创建管道按钮后，我收到以下错误信息：
作业创建失败
用户无法充当 Dataflow 工作服务帐户
请求 ID：38414979878078088000807058
为了创建管道，我需要哪些权限？
我拥有 Dataflow Admin 权限。
英文:
I am creating dataflow pipeline for Custom Apache Beam Template from GCP console for custom template I already uploaded to Google Cloud Storage.
1 After selecting template path, I am seeing this warning or may be error
Template metadata error details
Fail to process as Flex Template and Legacy Template. Flex Template Process result:No SDK language information is provided., Legacy Template Process result:(35197769719996987970000d1381234f): Unable to open template file: gs://abc/extraction/templates/cron-template_metadata.
I do have gs://abc/extraction/templates/cron-template because that's what i specified in my args --templateLocation=gs://abc/extraction/templates/cron-template file but not gs://abc/extraction/templates/cron-template_metadata.
I am confused if it is a warning or error.
2 After I hit button for creating the pipeline
I am getting this error
Job creation failed
User cannot act as Dataflow worker service account
Request ID: 38414979878078088000807058
What permission do I need in order to create a pipeline.
I have Dataflow Admin permission.
答案1
得分: 1
- Template metadata error details
Fail to process as Flex Template and Legacy Template. Flex Template Process result:No SDK language information is provided., Legacy Template Process result:(35197769719996987970000d1381234f): Unable to open template file: gs://abc/extraction/templates/cron-template_metadata.
This is an warning and can be ignored.
- User cannot act as Dataflow worker service account
We need to add a below permission to user who is creating the dataflow pipeline and this can be fixed
Service Account User
> Users granted the Service Account User role on a service account can
> use it to indirectly access all the resources to which the service
> account has access. For example, if a service account has been granted
> the Compute Admin role (roles/compute.admin), a user that has been
> granted the Service Account Users role (roles/iam.serviceAccountUser)
> on that service account can act as the service account to start a
> Compute Engine instance. In this flow, the user impersonates the
> service account to perform any tasks using its granted roles and
> permissions.
Reference : https://cloud.google.com/iam/docs/service-account-permissions#user-role
英文:
1 Template metadata error details
Fail to process as Flex Template and Legacy Template. Flex Template Process result:No SDK language information is provided., Legacy Template Process result:(35197769719996987970000d1381234f): Unable to open template file: gs://abc/extraction/templates/cron-template_metadata.
This is an warning and can be ignored.
2 User cannot act as Dataflow worker service account
We need to add a below permission to user who is creating the dataflow pipeline and this can be fixed
Service Account User
> Users granted the Service Account User role on a service account can
> use it to indirectly access all the resources to which the service
> account has access. For example, if a service account has been granted
> the Compute Admin role (roles/compute.admin), a user that has been
> granted the Service Account Users role (roles/iam.serviceAccountUser)
> on that service account can act as the service account to start a
> Compute Engine instance. In this flow, the user impersonates the
> service account to perform any tasks using its granted roles and
> permissions.
Reference : https://cloud.google.com/iam/docs/service-account-permissions#user-role
答案2
得分: 0
有关Flex模板的权限的更多信息：https://cloud.google.com/dataflow/docs/guides/templates/configuring-flex-templates#permissions
英文:
More about the permissions for Flex Templates: https://cloud.google.com/dataflow/docs/guides/templates/configuring-flex-templates#permissions
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论