为什么这个lambda函数没有复制S3对象?

huangapple go评论74阅读模式
英文:

Why is this lambda function not copying S3 objects?

问题

我刚刚创建了一个使用 Node.js 18.x 运行时的 Lambda 函数。它从一个存储桶中检索对象列表,但复制操作似乎不起作用,尽管返回的状态是“成功”。我还观察到,在复制命令之前的 console.log 输出,但在命令后的 console.log 却没有输出。

有人看到我可能忽略的明显问题吗?

import { S3Client, ListObjectsV2Command, CopyObjectCommand } from "@aws-sdk/client-s3"

const client = new S3Client({})

export const handler = async (event) => {
  const sourceBucket = 'in-bucket'
  const destBucket = 'complete-bucket'
  const dateSuffix = '-' + (new Date()).toISOString().substring(0,10).replaceAll('-','')
  const commandList = new ListObjectsV2Command({Bucket: sourceBucket})

  try {
    let isTruncated = true

    while (isTruncated) {
      const { Contents, IsTruncated, NextContinuationToken } = await client.send(commandList)
      Contents.forEach(async (c) => {
        const fn = c.Key
        const options = {
          CopySource: encodeURI(`${sourceBucket}/${fn}`),
          Bucket: destBucket,
          Key: encodeURI(`${fn}${dateSuffix}`)
        }
        console.log(`moving ${fn}`)
        const commandCopy = new CopyObjectCommand(options)
        let response = await client.send(commandCopy)
        console.log(`response: ${response}`)
      })
      isTruncated = IsTruncated
      commandList.input.ContinuationToken = NextContinuationToken
    }
    console.log('Finished')

  } catch (err) {
    console.error(err)
  }
}

执行结果:

Test Event Name
(unsaved) test event

Response
null

Function Logs
START RequestId: dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2 Version: $LATEST
2023-07-27T17:50:28.279Z	dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2	INFO	moving file01.csv
2023-07-27T17:50:28.281Z	dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2	INFO	moving file02.csv
2023-07-27T17:50:28.337Z	dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2	INFO	moving file03.csv
2023-07-27T17:50:28.338Z	dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2	INFO	moving file04.csv
2023-07-27T17:50:28.338Z	dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2	INFO	moving file05.csv
2023-07-27T17:50:28.338Z	dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2	INFO	moving file06.csv
2023-07-27T17:50:28.338Z	dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2	INFO	moving file07.csv
2023-07-27T17:50:28.339Z	dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2	INFO	moving file08.csv
2023-07-27T17:50:28.339Z	dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2	INFO	moving file09.csv
2023-07-27T17:50:28.340Z	dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2	INFO	moving file10.csv
2023-07-27T17:50:28.340Z	dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2	INFO	moving file11.csv
2023-07-27T17:50:28.340Z	dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2	INFO	moving file12.csv
2023-07-27T17:50:28.340Z	dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2	INFO	moving file13.csv
2023-07-27T17:50:28.341Z	dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2	INFO	moving file14.csv
2023-07-27T17:50:28.341Z	dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2	INFO	moving file15.csv
2023-07-27T17:50:28.397Z	dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2	INFO	moving file16.csv
2023-07-27T17:50:28.397Z	dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2	INFO	Finished
END RequestId: dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2
REPORT RequestId: dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2	Duration: 1178.43 ms	Billed Duration: 1179 ms	Memory Size: 128 MB	Max Memory Used: 103 MB	Init Duration: 621.73 ms

Request ID
dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2

我应用了示例代码,例如 https://docs.aws.amazon.com/AmazonS3/latest/userguide/example_s3_CopyObject_section.html
由于第一个命令返回了所有预期的键值,我期望第二个命令将对象从源存储桶复制到目标存储桶。

英文:

I've just created a lambda function with runtime Node.js 18.x.
It retrieves a list of S3 objects from one bucket, but the copy is not working, even though the status returned is 'Successful'. I've also observed that the console.log before the copy command is output, but not the console.log following the command.

Does anyone see anything obvious that I've overlooked?

import { S3Client, ListObjectsV2Command, CopyObjectCommand } from "@aws-sdk/client-s3"

const client = new S3Client({})

export const handler = async (event) => {
  const sourceBucket = 'in-bucket'
  const destBucket = 'complete-bucket'
  const dateSuffix = '-' + (new Date()).toISOString().substring(0,10).replaceAll('-','')
  const commandList = new ListObjectsV2Command({Bucket: sourceBucket})

  try {
    let isTruncated = true

    while (isTruncated) {
      const { Contents, IsTruncated, NextContinuationToken } = await client.send(commandList)
      Contents.forEach(async (c) => {
        const fn = c.Key
        const options = {
          CopySource: encodeURI(`${sourceBucket}/${fn}`),
          Bucket: destBucket,
          Key: encodeURI(`${fn}${dateSuffix}`)
        }
        console.log(`moving ${fn}`)
        const commandCopy = new CopyObjectCommand(options)
        let response = await client.send(commandCopy)
        console.log(`response: ${response}`)
      })
      isTruncated = IsTruncated
      commandList.input.ContinuationToken = NextContinuationToken
    }
    console.log('Finished')

  } catch (err) {
    console.error(err)
  }
}

Execution results:

Test Event Name
(unsaved) test event

Response
null

Function Logs
START RequestId: dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2 Version: $LATEST
2023-07-27T17:50:28.279Z	dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2	INFO	moving file01.csv
2023-07-27T17:50:28.281Z	dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2	INFO	moving file02.csv
2023-07-27T17:50:28.337Z	dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2	INFO	moving file03.csv
2023-07-27T17:50:28.338Z	dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2	INFO	moving file04.csv
2023-07-27T17:50:28.338Z	dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2	INFO	moving file05.csv
2023-07-27T17:50:28.338Z	dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2	INFO	moving file06.csv
2023-07-27T17:50:28.338Z	dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2	INFO	moving file07.csv
2023-07-27T17:50:28.339Z	dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2	INFO	moving file08.csv
2023-07-27T17:50:28.339Z	dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2	INFO	moving file09.csv
2023-07-27T17:50:28.340Z	dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2	INFO	moving file10.csv
2023-07-27T17:50:28.340Z	dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2	INFO	moving file11.csv
2023-07-27T17:50:28.340Z	dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2	INFO	moving file12.csv
2023-07-27T17:50:28.340Z	dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2	INFO	moving file13.csv
2023-07-27T17:50:28.341Z	dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2	INFO	moving file14.csv
2023-07-27T17:50:28.341Z	dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2	INFO	moving file15.csv
2023-07-27T17:50:28.397Z	dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2	INFO	moving file16.csv
2023-07-27T17:50:28.397Z	dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2	INFO	Finished
END RequestId: dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2
REPORT RequestId: dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2	Duration: 1178.43 ms	Billed Duration: 1179 ms	Memory Size: 128 MB	Max Memory Used: 103 MB	Init Duration: 621.73 ms

Request ID
dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2

I applied sample code, e.g., https://docs.aws.amazon.com/AmazonS3/latest/userguide/example_s3_CopyObject_section.html
Since the first command returned all the expected key values, I expected the second command to copy the objects from the source bucket to the destination bucket.

答案1

得分: 1

谢谢Ervin Szilagyi指出:

> 你使用了一个带有异步函数的forEach。这很可能不会按照你的期望工作,forEach循环和因此你的lambda函数很可能在任何复制发生之前就完成了。请参考这个Stackoverflow答案,了解在forEach中使用异步函数的用法:stackoverflow.com/a/70946414/766111

这正是问题所在。当我将:

Contents.forEach(async (c) => {

替换为:

for (const c of Contents) {

它按照预期工作。

英文:

Thank you, Ervin Szilagyi, for pointing out:

>You have a forEach with an async function. This most likely wont work as you might expect, the forEach loop, and hence your lambda function, will most likely finish before any copying is happening. See this Stackoverflow answer regarding usage of async function in forEach: stackoverflow.com/a/70946414/766111

That was exactly the issue. When I replaced:

Contents.forEach(async (c) => {

with:

for (const c of Contents) {

it worked as expected.

huangapple
  • 本文由 发表于 2023年7月28日 02:29:06
  • 转载请务必保留本文链接:https://go.coder-hub.com/76782523.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定