有问题通过boto3下载S3存储桶对象。错误403 HeadObject:禁止。

huangapple go评论69阅读模式
英文:

Trouble downloading S3 bucket objects through boto3. Error 403 HeadObject: Forbidden

问题

我知道这个问题还有其他帖子,但仍然在努力找到正确的解决方案。我试图使用以下Python脚本下载S3存储桶中的一组特定对象(我有访问权限)。运行脚本时,第一个对象成功下载,然后出现以下错误(403):

botocore.exceptions.ClientError: 调用 HeadObject 操作时发生错误 (403): 禁止访问

请看下面的代码:

import csv
import boto3
import re
import logging
from botocore.exceptions import ClientError

prod_number_array_bq = []
prod_number_array_s3 = []
with open('bq-results-20191218-151637-rshujisvqrri.csv') as csv_file:
    csv_reader = csv.reader(csv_file, delimiter=',')
    line_count = 0
    for row in csv_reader:
        sliced = re.sub("[^0-9]", "", str(row))
        prod_number_array_bq.append(sliced)

s3 = boto3.resource('s3')
bucket = s3.Bucket('********')

for key in bucket.objects.all():
    sliced = re.sub("[^0-9]", "", str(key.key))
    if (set(sliced) & set(prod_number_array_bq)) != "":
        bucket.download_file(key.key, sliced + '.txt')

希望能得到帮助 有问题通过boto3下载S3存储桶对象。错误403 HeadObject:禁止。

谢谢

英文:

I'm aware there are other threads on here about this issue but am still struggling to find the right solution. I am attempting to download a set of specific objects within an S3 bucket (that I do have access to) using the following python script. When running the script, the first object successfully downloads but then this error (403) is thrown:

botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden

See below my code:

import csv
import boto3
import re
import logging
from botocore.exceptions import ClientError

prod_number_array_bq = []
prod_number_array_s3 = []
with open('bq-results-20191218-151637-rshujisvqrri.csv') as csv_file:
	csv_reader = csv.reader(csv_file,delimiter=',')
	line_count = 0
	for row in csv_reader:
		sliced = re.sub("[^0-9]", "", str(row))
		prod_number_array_bq.append(sliced)

s3 = boto3.resource('s3')
bucket = s3.Bucket('********')

for key in bucket.objects.all():
	sliced = re.sub("[^0-9]", "", str(key.key))
	if((set(sliced) & set(prod_number_array_bq))!=""):
			bucket.download_file(key.key,sliced + '.txt')

Help would be appreciated 有问题通过boto3下载S3存储桶对象。错误403 HeadObject:禁止。

Thanks

答案1

得分: 7

通常情况下,即使具有s3:GetObject权限,但在HeadObject请求时收到403错误,原因是未为存储桶提供s3:ListObjects权限并且您的键不存在。这是一项安全措施,用于防止暴露存储桶中的对象信息。当您同时具有存储桶中对象的s3:GetObject权限和存储桶本身的s3:ListObjects权限时,对于不存在的键的响应是404 "未找到键"响应。如果您只具有s3:GetObject权限并请求不存在的对象,则响应是403 "访问被拒绝"。

英文:

Typically when you see a 403 on HeadObject despite having the s3:GetObject permission, it's because the s3:ListObjects permission wasn't provided for the bucket AND your key doesn't exist. It's a security measure to prevent exposing information about what objects are or aren't in your bucket. When you have both the s3:GetObject permission for the objects in a bucket, and the s3:ListObjects permission for the bucket itself, the response for a non-existent key is a 404 "no such key" response. If you only have s3:GetObject permission and request a non-existent object, the response is a 403 "access denied".

答案2

得分: 2

在我的情况下,我可以读取文件,但无法下载它。

所以以下代码会打印文件信息:

resp = s3_client.list_objects_v2(Bucket=bucket_name, Prefix=origin)
print(resp)

但然后会出现 botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden 错误。

问题是模型是从不同的AWS账户上传的。我们在上传时缺少ACL权限,所以,

所以我们使用以下命令上传了文件:

s3_client.upload_file(origin,
                      bucket_name,
                      destination,
                      ExtraArgs={'ACL':'bucket-owner-full-control'})

这使我们能够按照预期读取和下载文件。

英文:

In my case, I could read the file but couldn't download it

So I the following would have printed the file information

resp = s3_client.list_objects_v2(Bucket=bucket_name, Prefix=origin)
print(resp)

but then this would have given botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden error

s3_client.download_file(bucket_name, origin, destination) 

The problem was the model was uploaded from different AWS account. We were missing ACL on upload. so,

so we uploaded the file with the following command

s3_client.upload_file(origin,
                      bucket_name,
                      destination,
                      ExtraArgs={'ACL':'bucket-owner-full-control'})

and this led us to read and download the file as we expectd.

huangapple
  • 本文由 发表于 2020年1月7日 00:33:50
  • 转载请务必保留本文链接:https://go.coder-hub.com/59615713.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定