英文:
Trouble downloading S3 bucket objects through boto3. Error 403 HeadObject: Forbidden
问题
我知道这个问题还有其他帖子,但仍然在努力找到正确的解决方案。我试图使用以下Python脚本下载S3存储桶中的一组特定对象(我有访问权限)。运行脚本时,第一个对象成功下载,然后出现以下错误(403):
botocore.exceptions.ClientError: 调用 HeadObject 操作时发生错误 (403): 禁止访问
请看下面的代码:
import csv
import boto3
import re
import logging
from botocore.exceptions import ClientError
prod_number_array_bq = []
prod_number_array_s3 = []
with open('bq-results-20191218-151637-rshujisvqrri.csv') as csv_file:
csv_reader = csv.reader(csv_file, delimiter=',')
line_count = 0
for row in csv_reader:
sliced = re.sub("[^0-9]", "", str(row))
prod_number_array_bq.append(sliced)
s3 = boto3.resource('s3')
bucket = s3.Bucket('********')
for key in bucket.objects.all():
sliced = re.sub("[^0-9]", "", str(key.key))
if (set(sliced) & set(prod_number_array_bq)) != "":
bucket.download_file(key.key, sliced + '.txt')
希望能得到帮助
谢谢
英文:
I'm aware there are other threads on here about this issue but am still struggling to find the right solution. I am attempting to download a set of specific objects within an S3 bucket (that I do have access to) using the following python script. When running the script, the first object successfully downloads but then this error (403) is thrown:
botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden
See below my code:
import csv
import boto3
import re
import logging
from botocore.exceptions import ClientError
prod_number_array_bq = []
prod_number_array_s3 = []
with open('bq-results-20191218-151637-rshujisvqrri.csv') as csv_file:
csv_reader = csv.reader(csv_file,delimiter=',')
line_count = 0
for row in csv_reader:
sliced = re.sub("[^0-9]", "", str(row))
prod_number_array_bq.append(sliced)
s3 = boto3.resource('s3')
bucket = s3.Bucket('********')
for key in bucket.objects.all():
sliced = re.sub("[^0-9]", "", str(key.key))
if((set(sliced) & set(prod_number_array_bq))!=""):
bucket.download_file(key.key,sliced + '.txt')
Help would be appreciated
Thanks
答案1
得分: 7
通常情况下,即使具有s3:GetObject
权限,但在HeadObject
请求时收到403错误,原因是未为存储桶提供s3:ListObjects
权限并且您的键不存在。这是一项安全措施,用于防止暴露存储桶中的对象信息。当您同时具有存储桶中对象的s3:GetObject
权限和存储桶本身的s3:ListObjects
权限时,对于不存在的键的响应是404 "未找到键"响应。如果您只具有s3:GetObject
权限并请求不存在的对象,则响应是403 "访问被拒绝"。
英文:
Typically when you see a 403 on HeadObject
despite having the s3:GetObject
permission, it's because the s3:ListObjects
permission wasn't provided for the bucket AND your key doesn't exist. It's a security measure to prevent exposing information about what objects are or aren't in your bucket. When you have both the s3:GetObject
permission for the objects in a bucket, and the s3:ListObjects
permission for the bucket itself, the response for a non-existent key is a 404 "no such key" response. If you only have s3:GetObject
permission and request a non-existent object, the response is a 403 "access denied".
答案2
得分: 2
在我的情况下,我可以读取文件,但无法下载它。
所以以下代码会打印文件信息:
resp = s3_client.list_objects_v2(Bucket=bucket_name, Prefix=origin)
print(resp)
但然后会出现 botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden
错误。
问题是模型是从不同的AWS账户上传的。我们在上传时缺少ACL权限,所以,
所以我们使用以下命令上传了文件:
s3_client.upload_file(origin,
bucket_name,
destination,
ExtraArgs={'ACL':'bucket-owner-full-control'})
这使我们能够按照预期读取和下载文件。
英文:
In my case, I could read the file but couldn't download it
So I the following would have printed the file information
resp = s3_client.list_objects_v2(Bucket=bucket_name, Prefix=origin)
print(resp)
but then this would have given botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden
error
s3_client.download_file(bucket_name, origin, destination)
The problem was the model was uploaded from different AWS account. We were missing ACL on upload. so,
so we uploaded the file with the following command
s3_client.upload_file(origin,
bucket_name,
destination,
ExtraArgs={'ACL':'bucket-owner-full-control'})
and this led us to read and download the file as we expectd.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论