使用Golang和Lambda从S3读取文件时遇到问题。

huangapple go评论96阅读模式
英文:

problem reading file from s3 using golang with lambda

问题

我在golang中有以下代码,用于读取文件并提取文件的某些列,文件名是从我的lambda接收到的,但在获取来自s3的文件的第一步中,我遇到了问题。

我的代码如下:

package main

import (
	"fmt"
	"packageXXX/cmd/utils"

	"github.com/aws/aws-lambda-go/lambda"
	"github.com/aws/aws-sdk-go/aws"
	"github.com/aws/aws-sdk-go/aws/session"
	"github.com/aws/aws-sdk-go/service/s3"
	"github.com/aws/aws-sdk-go/service/s3/s3manager"
	"go.uber.org/zap"
)

var (
	log *zap.Logger
)

func init() {
	log = utils.InitLogger()
}

func DownloadFile(downloader *s3manager.Downloader, bucketName string, key string) error {

	buff := &aws.WriteAtBuffer{}

	numBytes, err := downloader.Download(
		buff,
		&s3.GetObjectInput{
			Bucket: aws.String(bucketName),
			Key:    aws.String(key),
		},
	)

	data := buff.Bytes() // 现在data是我的[]byte数组

	log.Info("nnnnnnnnnnnnnnn", zap.Any("nnn", numBytes))
	log.Info("xxxxxxxxxxxxxxx", zap.Any("bytes", data))
	return err
}

func handler(fileName string) (string, error) {
	log.Info(fmt.Sprintf("received: %s", fileName))
	sess, err := session.NewSessionWithOptions(session.Options{
		Profile: "default",
		Config: aws.Config{
			Region: aws.String("us-west-2"),
		},
	})

	if err != nil {
		log.Info("errorrrrrrrrr", zap.Any("session", err))
		return "errorr,", nil
	} else {
		log.Info("sessionnnnnnnnnnnnnn", zap.Any("sess", sess))
	}

	bucketName := "s3-test-lambda-go-xxx"
	downloader := s3manager.NewDownloader(sess)
	log.Info("downloader", zap.Any("downloader", err))
	key := fileName
	err = DownloadFile(downloader, bucketName, key)

	if err != nil {
		fmt.Printf("Couldn't download file: %v", err)
		return "errror", nil
	}

	fmt.Println("Successfully downloaded file")

	return fmt.Sprintf("el file recibido es: %s", fileName), nil
}

func main() {
	lambda.Start(handler)
}

打印的日志如下:

For verbose messaging see aws.Config.CredentialsChainVerboseErrorsSTART RequestId: dba8ebd5-9102-4a2f-8e45-cdef355f1bbd Version: $LATEST
{"level":"info","ts":"2022-11-02T17:31:32Z","msg":"received: gggggg2.txt;"}
{"level":"info","ts":"2022-11-02T17:31:32Z","msg":"sessionnnnnnnnnnnnnn","sessError":"json: unsupported type: endpoints.endpointDefaults"}
{"level":"info","ts":"2022-11-02T17:31:32Z","msg":"downloader","downloader":null}
{"level":"info","ts":"2022-11-02T17:31:38Z","msg":"nnnnnnnnnnnnnnn","nnn":0}
{"level":"info","ts":"2022-11-02T17:31:38Z","msg":"xxxxxxxxxxxxxxx","bytes":""}
Couldn't download file: NoCredentialProviders: no valid providers in chain. Deprecated.
END RequestId: dba8ebd5-9102-4a2f-8e45-cdef355f1bbd

问题出在session上。

那么,是否可以使用角色创建会话,因为在生产环境中,我不想使用访问密钥和秘密密钥?
非常感谢。

英文:

I have the following code in golang for read a file and extract some colums of the file, the fileName is received from my lambda, but in my first step that is get the file from s3 i am getting a problem.

My code is:

package main
import (
"fmt"
"packageXXX/cmd/utils"
"github.com/aws/aws-lambda-go/lambda"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/s3"
"github.com/aws/aws-sdk-go/service/s3/s3manager"
"go.uber.org/zap"
)
var (
log *zap.Logger
)
func init() {
log = utils.InitLogger()
}
func DownloadFile(downloader *s3manager.Downloader, bucketName string, key string) error {
buff := &aws.WriteAtBuffer{}
numBytes, err := downloader.Download(
buff,
&s3.GetObjectInput{
Bucket: aws.String(bucketName),
Key:    aws.String(key),
},
)
data := buff.Bytes() // now data is my []byte array
log.Info("nnnnnnnnnnnnnnn", zap.Any("nnn", numBytes))
log.Info("xxxxxxxxxxxxxxx", zap.Any("bytes", data))
return err
}
func handler(fileName string) (string, error) {
log.Info(fmt.Sprintf("received: %s", fileName))
sess, err := session.NewSessionWithOptions(session.Options{
Profile: "default",
Config: aws.Config{
Region: aws.String("us-west-2"),
},
})
if err != nil {
log.Info("errorrrrrrrrr", zap.Any("session", err))
return "errorr,", nil
} else {
log.Info("sessionnnnnnnnnnnnnn", zap.Any("sess", sess))
}
bucketName := "s3-test-lambda-go-xxx"
downloader := s3manager.NewDownloader(sess)
log.Info("downloader", zap.Any("downloader", err))
key := fileName
err = DownloadFile(downloader, bucketName, key)
if err != nil {
fmt.Printf("Couldn't download file: %v", err)
return "errror", nil
}
fmt.Println("Successfully downloaded file")
return fmt.Sprintf("el file recibido es: %s", fileName), nil
}
func main() {
lambda.Start(handler)
}

And the log that is printing is:

For verbose messaging see aws.Config.CredentialsChainVerboseErrorsSTART RequestId: dba8ebd5-9102-4a2f-8e45-cdef355f1bbd Version: $LATEST
{"level":"info","ts":"2022-11-02T17:31:32Z","msg":"received: gggggg2.txt"}
{"level":"info","ts":"2022-11-02T17:31:32Z","msg":"sessionnnnnnnnnnnnnn","sessError":"json: unsupported type: endpoints.endpointDefaults"}
{"level":"info","ts":"2022-11-02T17:31:32Z","msg":"downloader","downloader":null}
{"level":"info","ts":"2022-11-02T17:31:38Z","msg":"nnnnnnnnnnnnnnn","nnn":0}
{"level":"info","ts":"2022-11-02T17:31:38Z","msg":"xxxxxxxxxxxxxxx","bytes":""}
Couldn't download file: NoCredentialProviders: no valid providers in chain. Deprecated.
END RequestId: dba8ebd5-9102-4a2f-8e45-cdef355f1bbd

The problem is the session

Then Is possible create session with role, because like is production i don't want use access and secret key?
Thanks a lot

答案1

得分: 1

如果你像这样设置,会加载默认会话:

sess, err := session.NewSession(&aws.Config{
  Region: aws.String("us-east-1")},
)

而不是:

sess, err := session.NewSessionWithOptions(session.Options{
    Profile: "default",
    Config: aws.Config{
        Region: aws.String("us-west-2"),
    },
})

这意味着它将使用你配置的任何会话,不管你是在 Lambda 函数内部还是本地环境。这也意味着你可以假设任何你想要假设的角色,甚至可以使用 SSO 登录(因此不需要访问密钥和密钥对)。

英文:

Session will load your default session if you but it like this

sess, err := session.NewSession(&aws.Config{
  Region: aws.String("us-east-1")},
)

instead of:

    sess, err := session.NewSessionWithOptions(session.Options{
        Profile: "default",
        Config: aws.Config{
            Region: aws.String("us-west-2"),
        },
    })

This means it will use whatever session you have configured not depending on whether you are inside the lambda or local. This also means you can assume any role you want to assume or even use SSO to login (so no access key and secret are required).

huangapple
  • 本文由 发表于 2022年11月3日 01:44:00
  • 转载请务必保留本文链接:https://go.coder-hub.com/74293569.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定