英文:
Why is this lambda function not copying S3 objects?
问题
我刚刚创建了一个使用 Node.js 18.x 运行时的 Lambda 函数。它从一个存储桶中检索对象列表,但复制操作似乎不起作用,尽管返回的状态是“成功”。我还观察到,在复制命令之前的 console.log 输出,但在命令后的 console.log 却没有输出。
有人看到我可能忽略的明显问题吗?
import { S3Client, ListObjectsV2Command, CopyObjectCommand } from "@aws-sdk/client-s3"
const client = new S3Client({})
export const handler = async (event) => {
const sourceBucket = 'in-bucket'
const destBucket = 'complete-bucket'
const dateSuffix = '-' + (new Date()).toISOString().substring(0,10).replaceAll('-','')
const commandList = new ListObjectsV2Command({Bucket: sourceBucket})
try {
let isTruncated = true
while (isTruncated) {
const { Contents, IsTruncated, NextContinuationToken } = await client.send(commandList)
Contents.forEach(async (c) => {
const fn = c.Key
const options = {
CopySource: encodeURI(`${sourceBucket}/${fn}`),
Bucket: destBucket,
Key: encodeURI(`${fn}${dateSuffix}`)
}
console.log(`moving ${fn}`)
const commandCopy = new CopyObjectCommand(options)
let response = await client.send(commandCopy)
console.log(`response: ${response}`)
})
isTruncated = IsTruncated
commandList.input.ContinuationToken = NextContinuationToken
}
console.log('Finished')
} catch (err) {
console.error(err)
}
}
执行结果:
Test Event Name
(unsaved) test event
Response
null
Function Logs
START RequestId: dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2 Version: $LATEST
2023-07-27T17:50:28.279Z dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2 INFO moving file01.csv
2023-07-27T17:50:28.281Z dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2 INFO moving file02.csv
2023-07-27T17:50:28.337Z dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2 INFO moving file03.csv
2023-07-27T17:50:28.338Z dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2 INFO moving file04.csv
2023-07-27T17:50:28.338Z dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2 INFO moving file05.csv
2023-07-27T17:50:28.338Z dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2 INFO moving file06.csv
2023-07-27T17:50:28.338Z dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2 INFO moving file07.csv
2023-07-27T17:50:28.339Z dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2 INFO moving file08.csv
2023-07-27T17:50:28.339Z dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2 INFO moving file09.csv
2023-07-27T17:50:28.340Z dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2 INFO moving file10.csv
2023-07-27T17:50:28.340Z dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2 INFO moving file11.csv
2023-07-27T17:50:28.340Z dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2 INFO moving file12.csv
2023-07-27T17:50:28.340Z dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2 INFO moving file13.csv
2023-07-27T17:50:28.341Z dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2 INFO moving file14.csv
2023-07-27T17:50:28.341Z dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2 INFO moving file15.csv
2023-07-27T17:50:28.397Z dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2 INFO moving file16.csv
2023-07-27T17:50:28.397Z dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2 INFO Finished
END RequestId: dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2
REPORT RequestId: dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2 Duration: 1178.43 ms Billed Duration: 1179 ms Memory Size: 128 MB Max Memory Used: 103 MB Init Duration: 621.73 ms
Request ID
dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2
我应用了示例代码,例如 https://docs.aws.amazon.com/AmazonS3/latest/userguide/example_s3_CopyObject_section.html 。
由于第一个命令返回了所有预期的键值,我期望第二个命令将对象从源存储桶复制到目标存储桶。
英文:
I've just created a lambda function with runtime Node.js 18.x.
It retrieves a list of S3 objects from one bucket, but the copy is not working, even though the status returned is 'Successful'. I've also observed that the console.log before the copy command is output, but not the console.log following the command.
Does anyone see anything obvious that I've overlooked?
import { S3Client, ListObjectsV2Command, CopyObjectCommand } from "@aws-sdk/client-s3"
const client = new S3Client({})
export const handler = async (event) => {
const sourceBucket = 'in-bucket'
const destBucket = 'complete-bucket'
const dateSuffix = '-' + (new Date()).toISOString().substring(0,10).replaceAll('-','')
const commandList = new ListObjectsV2Command({Bucket: sourceBucket})
try {
let isTruncated = true
while (isTruncated) {
const { Contents, IsTruncated, NextContinuationToken } = await client.send(commandList)
Contents.forEach(async (c) => {
const fn = c.Key
const options = {
CopySource: encodeURI(`${sourceBucket}/${fn}`),
Bucket: destBucket,
Key: encodeURI(`${fn}${dateSuffix}`)
}
console.log(`moving ${fn}`)
const commandCopy = new CopyObjectCommand(options)
let response = await client.send(commandCopy)
console.log(`response: ${response}`)
})
isTruncated = IsTruncated
commandList.input.ContinuationToken = NextContinuationToken
}
console.log('Finished')
} catch (err) {
console.error(err)
}
}
Execution results:
Test Event Name
(unsaved) test event
Response
null
Function Logs
START RequestId: dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2 Version: $LATEST
2023-07-27T17:50:28.279Z dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2 INFO moving file01.csv
2023-07-27T17:50:28.281Z dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2 INFO moving file02.csv
2023-07-27T17:50:28.337Z dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2 INFO moving file03.csv
2023-07-27T17:50:28.338Z dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2 INFO moving file04.csv
2023-07-27T17:50:28.338Z dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2 INFO moving file05.csv
2023-07-27T17:50:28.338Z dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2 INFO moving file06.csv
2023-07-27T17:50:28.338Z dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2 INFO moving file07.csv
2023-07-27T17:50:28.339Z dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2 INFO moving file08.csv
2023-07-27T17:50:28.339Z dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2 INFO moving file09.csv
2023-07-27T17:50:28.340Z dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2 INFO moving file10.csv
2023-07-27T17:50:28.340Z dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2 INFO moving file11.csv
2023-07-27T17:50:28.340Z dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2 INFO moving file12.csv
2023-07-27T17:50:28.340Z dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2 INFO moving file13.csv
2023-07-27T17:50:28.341Z dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2 INFO moving file14.csv
2023-07-27T17:50:28.341Z dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2 INFO moving file15.csv
2023-07-27T17:50:28.397Z dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2 INFO moving file16.csv
2023-07-27T17:50:28.397Z dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2 INFO Finished
END RequestId: dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2
REPORT RequestId: dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2 Duration: 1178.43 ms Billed Duration: 1179 ms Memory Size: 128 MB Max Memory Used: 103 MB Init Duration: 621.73 ms
Request ID
dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2
I applied sample code, e.g., https://docs.aws.amazon.com/AmazonS3/latest/userguide/example_s3_CopyObject_section.html
Since the first command returned all the expected key values, I expected the second command to copy the objects from the source bucket to the destination bucket.
答案1
得分: 1
谢谢Ervin Szilagyi指出:
> 你使用了一个带有异步函数的forEach。这很可能不会按照你的期望工作,forEach循环和因此你的lambda函数很可能在任何复制发生之前就完成了。请参考这个Stackoverflow答案,了解在forEach
中使用异步函数的用法:stackoverflow.com/a/70946414/766111
这正是问题所在。当我将:
Contents.forEach(async (c) => {
替换为:
for (const c of Contents) {
它按照预期工作。
英文:
Thank you, Ervin Szilagyi, for pointing out:
>You have a forEach with an async function. This most likely wont work as you might expect, the forEach loop, and hence your lambda function, will most likely finish before any copying is happening. See this Stackoverflow answer regarding usage of async function in forEach
: stackoverflow.com/a/70946414/766111
That was exactly the issue. When I replaced:
Contents.forEach(async (c) => {
with:
for (const c of Contents) {
it worked as expected.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论