英文:

Issue with Work Identity Federation (WIF) and gsutil ACL set Command

问题

I've an existing setup in which I'm using workload identity federation (WIF) to authenticate circleCI with GCP and everything has been working perfectly fine. Its a simple workflow which uses gsutil -m rsync -d -r folder/ gs://bucket command to sync a folder with a GCS bucket.

我已经有一个现有的设置，其中我使用工作负载身份联合（WIF）来验证CircleCI与GCP，一切都运行得非常完美。这是一个简单的工作流程，使用gsutil -m rsync -d -r folder/ gs://bucket命令将文件夹与GCS存储桶同步。

I recently modified my workflow to also run following additional command right after rsync which is basically intended to mark all objects in the bucket as public. I know I can instead mark the whole bucket as public but without going into much details there is a specific reason I'm doing it this way.

我最近修改了我的工作流程，以在rsync之后运行以下额外的命令，基本上是为了将存储桶中的所有对象标记为公共。我知道我可以将整个存储桶标记为公共，但不详细介绍，我之所以这样做有特定的原因。

gsutil -m acl set -R -a public-read gs://bucket

after making above change, I see following error in circleCI when this command is executed:

在进行以上更改后，当执行此命令时，我在circleCI中看到以下错误：

Traceback (most recent call last):
  File "/usr/lib/google-cloud-sdk/platform/gsutil/gsutil", line 21, in <module>
    gsutil.RunMain()
  File "/usr/lib/google-cloud-sdk/platform/gsutil/gsutil.py", line 151, in RunMain
    sys.exit(gslib.__main__.main())
  ...

the error message is confusing since it says its an authentication issue but I know authentication is not the issue since the rsync command right before this is working fine.

错误消息令人困惑，因为它说这是一个身份验证问题，但我知道身份验证不是问题，因为在此之前的rsync命令正常工作。

I also modified the workflow to do gCloud auth list before ACL set and that command also shows that gCloud is authenticated.

我还修改了工作流程，在ACL设置之前执行gCloud auth list，该命令也显示gCloud已经进行了身份验证。

I know its also not an authorization issue since the service account which is being used by WIF has "Storage Admin" and "Storage Object Admin" roles which adds storage.buckets.* and storage.objects.* permissions to the account so it has already more than the required permissions needed to set public ACL on the bucket or its objects, you can verify what permission are required for gsutil acl set vs what are already there using following documentation links.

我知道这也不是授权问题，因为WIF使用的服务帐户具有“Storage Admin”和“Storage Object Admin”角色，为该帐户添加了storage.buckets.*和storage.objects.*权限，因此它已经具有设置存储桶或其对象的公共ACL所需的权限，您可以使用以下文档链接验证gsutil acl set所需的权限与已有的权限有何不同。

https://cloud.google.com/storage/docs/access-control/iam-gsutil

https://cloud.google.com/storage/docs/access-control/iam-roles

Surprisingly if I remove WIF authentication and If I directly use a service account key for authentication the error goes away and gsutil acl set works fine, which tells me there might be an issue with WIF configuration but nothing seems out of the ordinary to me. I followed [this blog post by circleCI][1] to setup OIDC authentication / WIF for GCP. The issue seems to be specific to the gsutil acl set command, as other gsutil commands (like rsync) are working fine with WIF authentication. I don't want to use service account keys for authentication since google recommends against using them as they can pose security risk if compromised.

令人惊讶的是，如果我删除WIF身份验证，如果我直接使用服务帐户密钥进行身份验证，错误就会消失，gsutil acl set正常工作，这告诉我可能存在WIF配置问题，但在我看来没有什么不寻常的地方。我按照[CircleCI的这篇博文][1]设置OIDC身份验证/WIF用于GCP。问题似乎特定于gsutil acl set命令，因为其他gsutil命令（如rsync）在使用WIF身份验证时工作正常。我不想使用服务帐户密钥进行身份验证，因为Google建议不要使用它们，因为如果泄露可能会带来安全风险。

What I've tried so far,

到目前为止，我尝试过以下方法：

1. I tried to use private ACL instead of public-read just to make sure its not an issue specific to specific ACL
   我尝试使用private ACL而不是public-read，只是为了确保这不是特定ACL的问题

2. I've verified the gsutil version, and circleCI is using the latest version i.e. 5.24
   我已验证gsutil的版本，circleCI正在使用最新版本，即5.24

3. I tried removing the -m flag which is used for multi threading and is known to cause issue sometimes
   我尝试删除用于多线程的-m标志，因为它有时会引发问题

4. I checked GCP logs explorer if there are any additional logs available but there are none
   我检查了GCP日志资源管理器，看是否有其他可用的日志，但没有找到

5. I Enabled debug logging for gsutil / boto3 to see if it logs some additional info by doing following before the acl set command, but it didn't log any additional information,
   我启用了gsutil / boto3的调试日志，以查看是否在执行acl设置命令之前记录了一些额外信息，但没有记录任何额外信息，

export BOTO_CONFIG=/home/circleci/.boto/boto.cfg
echo "[Boto]" >> $BOTO_CONFIG
echo "debug =

<details>
<summary>英文:</summary>

I&#39;ve an existing setup in which I&#39;m using workload identity federation (WIF) to authenticate circleCI with GCP and everything has been working perfectly fine. Its a simple workflow which uses `gsutil -m rsync -d -r folder/ gs://bucket` command to sync a folder with a GCS bucket.

I recently modified my workflow to also run following additional command right after rsync which is basically intended to mark all objects in the bucket as public. I know I can instead mark the whole bucket as public but without going into much details there is a specific reason I&#39;m doing it this way.

`gsutil -m acl set -R -a public-read gs://bucket`

after making above change, I see following error in circleCI when this command is executed:

    Traceback (most recent call last):
      File &quot;/usr/lib/google-cloud-sdk/platform/gsutil/gsutil&quot;, line 21, in &lt;module&gt;
        gsutil.RunMain()
      File &quot;/usr/lib/google-cloud-sdk/platform/gsutil/gsutil.py&quot;, line 151, in RunMain
        sys.exit(gslib.__main__.main())
      File &quot;/usr/lib/google-cloud-sdk/platform/gsutil/gslib/__main__.py&quot;, line 436, in main
        return _RunNamedCommandAndHandleExceptions(
      File &quot;/usr/lib/google-cloud-sdk/platform/gsutil/gslib/__main__.py&quot;, line 785, in _RunNamedCommandAndHandleExceptions
        _HandleUnknownFailure(e)
      File &quot;/usr/lib/google-cloud-sdk/platform/gsutil/gslib/__main__.py&quot;, line 633, in _RunNamedCommandAndHandleExceptions
        return command_runner.RunNamedCommand(command_name,
      File &quot;/usr/lib/google-cloud-sdk/platform/gsutil/gslib/command_runner.py&quot;, line 421, in RunNamedCommand
        return_code = command_inst.RunCommand()
      File &quot;/usr/lib/google-cloud-sdk/platform/gsutil/gslib/commands/acl.py&quot;, line 587, in RunCommand
        self._SetAcl()
      File &quot;/usr/lib/google-cloud-sdk/platform/gsutil/gslib/commands/acl.py&quot;, line 411, in _SetAcl
        self.SetAclCommandHelper(SetAclFuncWrapper, SetAclExceptionHandler)
      File &quot;/usr/lib/google-cloud-sdk/platform/gsutil/gslib/command.py&quot;, line 1114, in SetAclCommandHelper
        canned_acls = storage_uri.canned_acls()
      File &quot;/usr/lib/google-cloud-sdk/platform/gsutil/gslib/vendored/boto/boto/storage_uri.py&quot;, line 220, in canned_acls
        conn = self.connect()
      File &quot;/usr/lib/google-cloud-sdk/platform/gsutil/gslib/vendored/boto/boto/storage_uri.py&quot;, line 121, in connect
        self.connection = GSConnection(access_key_id,
      File &quot;/usr/lib/google-cloud-sdk/platform/gsutil/gslib/vendored/boto/boto/gs/connection.py&quot;, line 45, in __init__
        super(GSConnection, self).__init__(gs_access_key_id, gs_secret_access_key,
      File &quot;/usr/lib/google-cloud-sdk/platform/gsutil/gslib/vendored/boto/boto/s3/connection.py&quot;, line 202, in __init__
        super(S3Connection, self).__init__(host,
      File &quot;/usr/lib/google-cloud-sdk/platform/gsutil/gslib/vendored/boto/boto/connection.py&quot;, line 572, in __init__
        self._auth_handler = auth.get_auth_handler(
      File &quot;/usr/lib/google-cloud-sdk/platform/gsutil/gslib/vendored/boto/boto/auth.py&quot;, line 1033, in get_auth_handler
        raise boto.exception.NoAuthHandlerFound(
    boto.exception.NoAuthHandlerFound: No handler was ready to authenticate. 4 handlers were checked. [&#39;DevshellAuth&#39;, &#39;HmacAuthV1Handler&#39;, &#39;OAuth2Auth&#39;, &#39;OAuth2ServiceAccountAuth&#39;] Check your credentials


the error message is confusing since it says its an authentication issue but I know authentication is not the issue since the rsync command right before this is working fine. I also modified the workflow to do `gCloud auth list` before ACL set and that command also shows that gCloud is authenticated.

I know its also not an authorization issue since the service account which is being used by WIF has &quot;Storage Admin&quot; and &quot;Storage Object Admin&quot; roles which adds storage.buckets.* and storage.objects.* permissions to the account so it has already more than the required permissions needed to set public ACL on the bucket or its objects, you can verify what permission are required for `gsutil acl set` vs what are already there using following documentation links.

https://cloud.google.com/storage/docs/access-control/iam-gsutil

https://cloud.google.com/storage/docs/access-control/iam-roles

Surprisingly if I remove WIF authentication and If I directly use a service account key for authentication the error goes away and `gsutil acl set` works fine, which tells me there might be an issue with WIF configuration but nothing seems out of the ordinary to me. I followed [this blog post by circleCI][1] to setup OIDC authentication / WIF for GCP. The issue seems to be specific to the `gsutil acl set` command, as other gsutil commands (like rsync) are working fine with WIF authentication. I don&#39;t want to use service account keys for authentication since google recommends against using them as they can pose security risk if compromised.

What I&#39;ve tried so far,

 1. I tried to use `private` ACL instead of `public-read` just to make sure its not an issue specific to specific ACL
 2. I&#39;ve verified the gsutil version, and circleCI is using the latest version i.e. 5.24
 3. I tried removing the -m flag which is used for multi threading and is known to cause issue sometimes
 4. I checked GCP logs explorer if there are any additional logs available but there are none
 5. I Enabled debug logging for gsutil / boto3 to see if it logs some additional info by doing following before the acl set command, but it didn&#39;t log any additional information,
 

export BOTO_CONFIG=/home/circleci/.boto/boto.cfg
echo "[Boto]" >> $BOTO_CONFIG
echo "debug = 2" >> $BOTO_CONFIG
gsutil acl set -R -a public-read gs://bucket

 
 6. I've checked if there are diff scopes available when using service account key vs WIF, by using following curl but both showed same scopes,

access_token=$(gcloud auth print-access-token) && curl "https://oauth2.googleapis.com/tokeninfo?access_token=${access_token}"

any help in this regard is welcome.

  [1]: https://circleci.com/blog/openid-connect-identity-tokens/

</details>


# 答案1
**得分**: 0

以下是翻译好的部分：

原文:
So it turns out the issue was happening when boto was trying to make a connection to GCP in order to resolve canned ACL i.e. `public-read`. Not sure why this happens since this is for the gsutils or boto developers to check but as as a workaround I switched to the `ch` sub-command, as that allows for setting ACLs without using canned ACLs. following is the alternate gsutil command which works fine with WIF.

翻译:
问题出现在当boto试图连接到GCP以解析canned ACL，即`public-read`时。不确定为什么会发生这种情况，因为这是供gsutils或boto开发人员检查的，但作为一种解决方法，我切换到了`ch`子命令，因为它允许在不使用canned ACL的情况下设置ACL。以下是使用WIF正常工作的替代gsutil命令。

<details>
<summary>英文:</summary>

So it turns out the issue was happening when boto was trying to make a connection to GCP in order to resolve canned ACL i.e. `public-read`. Not sure why this happens since this is for the gsutils or boto developers to check but as as a workaround I switched to the `ch` sub-command, as that allows for setting ACLs without using canned ACLs. following is the alternate gsutil command which works fine with WIF.

    gsutil -m acl ch -u AllUsers:READ gs://bucket/**

</details>