英文:
problem reading file from s3 using golang with lambda
问题
我在golang中有以下代码,用于读取文件并提取文件的某些列,文件名是从我的lambda接收到的,但在获取来自s3的文件的第一步中,我遇到了问题。
我的代码如下:
package main
import (
"fmt"
"packageXXX/cmd/utils"
"github.com/aws/aws-lambda-go/lambda"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/s3"
"github.com/aws/aws-sdk-go/service/s3/s3manager"
"go.uber.org/zap"
)
var (
log *zap.Logger
)
func init() {
log = utils.InitLogger()
}
func DownloadFile(downloader *s3manager.Downloader, bucketName string, key string) error {
buff := &aws.WriteAtBuffer{}
numBytes, err := downloader.Download(
buff,
&s3.GetObjectInput{
Bucket: aws.String(bucketName),
Key: aws.String(key),
},
)
data := buff.Bytes() // 现在data是我的[]byte数组
log.Info("nnnnnnnnnnnnnnn", zap.Any("nnn", numBytes))
log.Info("xxxxxxxxxxxxxxx", zap.Any("bytes", data))
return err
}
func handler(fileName string) (string, error) {
log.Info(fmt.Sprintf("received: %s", fileName))
sess, err := session.NewSessionWithOptions(session.Options{
Profile: "default",
Config: aws.Config{
Region: aws.String("us-west-2"),
},
})
if err != nil {
log.Info("errorrrrrrrrr", zap.Any("session", err))
return "errorr,", nil
} else {
log.Info("sessionnnnnnnnnnnnnn", zap.Any("sess", sess))
}
bucketName := "s3-test-lambda-go-xxx"
downloader := s3manager.NewDownloader(sess)
log.Info("downloader", zap.Any("downloader", err))
key := fileName
err = DownloadFile(downloader, bucketName, key)
if err != nil {
fmt.Printf("Couldn't download file: %v", err)
return "errror", nil
}
fmt.Println("Successfully downloaded file")
return fmt.Sprintf("el file recibido es: %s", fileName), nil
}
func main() {
lambda.Start(handler)
}
打印的日志如下:
For verbose messaging see aws.Config.CredentialsChainVerboseErrorsSTART RequestId: dba8ebd5-9102-4a2f-8e45-cdef355f1bbd Version: $LATEST
{"level":"info","ts":"2022-11-02T17:31:32Z","msg":"received: gggggg2.txt;"}
{"level":"info","ts":"2022-11-02T17:31:32Z","msg":"sessionnnnnnnnnnnnnn","sessError":"json: unsupported type: endpoints.endpointDefaults"}
{"level":"info","ts":"2022-11-02T17:31:32Z","msg":"downloader","downloader":null}
{"level":"info","ts":"2022-11-02T17:31:38Z","msg":"nnnnnnnnnnnnnnn","nnn":0}
{"level":"info","ts":"2022-11-02T17:31:38Z","msg":"xxxxxxxxxxxxxxx","bytes":""}
Couldn't download file: NoCredentialProviders: no valid providers in chain. Deprecated.
END RequestId: dba8ebd5-9102-4a2f-8e45-cdef355f1bbd
问题出在session上。
那么,是否可以使用角色创建会话,因为在生产环境中,我不想使用访问密钥和秘密密钥?
非常感谢。
英文:
I have the following code in golang for read a file and extract some colums of the file, the fileName is received from my lambda, but in my first step that is get the file from s3 i am getting a problem.
My code is:
package main
import (
"fmt"
"packageXXX/cmd/utils"
"github.com/aws/aws-lambda-go/lambda"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/s3"
"github.com/aws/aws-sdk-go/service/s3/s3manager"
"go.uber.org/zap"
)
var (
log *zap.Logger
)
func init() {
log = utils.InitLogger()
}
func DownloadFile(downloader *s3manager.Downloader, bucketName string, key string) error {
buff := &aws.WriteAtBuffer{}
numBytes, err := downloader.Download(
buff,
&s3.GetObjectInput{
Bucket: aws.String(bucketName),
Key: aws.String(key),
},
)
data := buff.Bytes() // now data is my []byte array
log.Info("nnnnnnnnnnnnnnn", zap.Any("nnn", numBytes))
log.Info("xxxxxxxxxxxxxxx", zap.Any("bytes", data))
return err
}
func handler(fileName string) (string, error) {
log.Info(fmt.Sprintf("received: %s", fileName))
sess, err := session.NewSessionWithOptions(session.Options{
Profile: "default",
Config: aws.Config{
Region: aws.String("us-west-2"),
},
})
if err != nil {
log.Info("errorrrrrrrrr", zap.Any("session", err))
return "errorr,", nil
} else {
log.Info("sessionnnnnnnnnnnnnn", zap.Any("sess", sess))
}
bucketName := "s3-test-lambda-go-xxx"
downloader := s3manager.NewDownloader(sess)
log.Info("downloader", zap.Any("downloader", err))
key := fileName
err = DownloadFile(downloader, bucketName, key)
if err != nil {
fmt.Printf("Couldn't download file: %v", err)
return "errror", nil
}
fmt.Println("Successfully downloaded file")
return fmt.Sprintf("el file recibido es: %s", fileName), nil
}
func main() {
lambda.Start(handler)
}
And the log that is printing is:
For verbose messaging see aws.Config.CredentialsChainVerboseErrorsSTART RequestId: dba8ebd5-9102-4a2f-8e45-cdef355f1bbd Version: $LATEST
{"level":"info","ts":"2022-11-02T17:31:32Z","msg":"received: gggggg2.txt"}
{"level":"info","ts":"2022-11-02T17:31:32Z","msg":"sessionnnnnnnnnnnnnn","sessError":"json: unsupported type: endpoints.endpointDefaults"}
{"level":"info","ts":"2022-11-02T17:31:32Z","msg":"downloader","downloader":null}
{"level":"info","ts":"2022-11-02T17:31:38Z","msg":"nnnnnnnnnnnnnnn","nnn":0}
{"level":"info","ts":"2022-11-02T17:31:38Z","msg":"xxxxxxxxxxxxxxx","bytes":""}
Couldn't download file: NoCredentialProviders: no valid providers in chain. Deprecated.
END RequestId: dba8ebd5-9102-4a2f-8e45-cdef355f1bbd
The problem is the session
Then Is possible create session with role, because like is production i don't want use access and secret key?
Thanks a lot
答案1
得分: 1
如果你像这样设置,会加载默认会话:
sess, err := session.NewSession(&aws.Config{
Region: aws.String("us-east-1")},
)
而不是:
sess, err := session.NewSessionWithOptions(session.Options{
Profile: "default",
Config: aws.Config{
Region: aws.String("us-west-2"),
},
})
这意味着它将使用你配置的任何会话,不管你是在 Lambda 函数内部还是本地环境。这也意味着你可以假设任何你想要假设的角色,甚至可以使用 SSO 登录(因此不需要访问密钥和密钥对)。
英文:
Session will load your default session if you but it like this
sess, err := session.NewSession(&aws.Config{
Region: aws.String("us-east-1")},
)
instead of:
sess, err := session.NewSessionWithOptions(session.Options{
Profile: "default",
Config: aws.Config{
Region: aws.String("us-west-2"),
},
})
This means it will use whatever session you have configured not depending on whether you are inside the lambda or local. This also means you can assume any role you want to assume or even use SSO to login (so no access key and secret are required).
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论