英文:
Golang IO reading file invalid memory address
问题
我正在尝试从AWS S3中读取一个GZIP文件时遇到一些问题。
我只有一个简单的代码,但是我得到了错误panic: runtime error: invalid memory address or nil pointer dereference
。但是很难找到问题所在。
错误消息说的是gzip.NewReader...
。为什么golang
会报告这个错误?我该如何解决它?
主要代码(pushego.go):
package main
import (
"bufio"
"flag"
"log"
"os"
"time"
"github.com/hensg/pushego/aws"
)
func init() {
log.SetOutput(os.Stdout)
}
func main() {
log.Println("Starting...")
var bucket, key string
var timeout time.Duration
flag.StringVar(&bucket, "b", "", "Bucket name")
flag.StringVar(&key, "k", "", "Object key name")
flag.DurationVar(&timeout, "d", 0, "Download timeout")
flag.Parse()
gzipReader, err := aws.GetGZIPDump(bucket, key, timeout) // line 28
if err != nil {
log.Fatal("Failed to create GZIP reader")
}
defer gzipReader.Close()
scanner := bufio.NewScanner(gzipReader)
for scanner.Scan() {
log.Printf("Read: %s\n", scanner.Text())
}
log.Printf("Successfully download file from %s/%s\n", bucket, key)
}
AWS代码(aws/s3.go):
package aws
import (
"compress/gzip"
"context"
"log"
"time"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/awserr"
"github.com/aws/aws-sdk-go/aws/request"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/s3"
)
func GetGZIPProductsDump(bucket, key string, timeout time.Duration) (*gzip.Reader, error) {
sess := session.Must(session.NewSession())
svc := s3.New(sess)
// Create a context with a timeout that will abort the download if it takes
// more than the passed in timeout.
ctx := context.Background()
var cancelFn func()
if timeout > 0 {
ctx, cancelFn = context.WithTimeout(ctx, timeout)
}
// Ensure the context is canceled to prevent leaking.
// See context package for more information, https://golang.org/pkg/context/
defer cancelFn()
// Uploads the object to S3. The Context will interrupt the request if the
// timeout expires.
resp, err := svc.GetObjectWithContext(ctx, &s3.GetObjectInput{
Bucket: aws.String(bucket),
Key: aws.String(key),
})
if err != nil {
if aerr, ok := err.(awserr.Error); ok && aerr.Code() == request.CanceledErrorCode {
// If the SDK can determine the request or retry delay was canceled
// by a context the CanceledErrorCode error code will be returned.
log.Fatal("Download canceled due to timeout, %v\n", err)
} else {
log.Fatal("Failed to download object, %v\n", err)
}
}
return gzip.NewReader(resp.Body) // line 47
}
希望这可以帮助你解决问题。
英文:
I'm having some troubles trying to read a GZIP file from AWS S3.
I have only a simple code and I'm getting the error panic: runtime error: invalid memory address or nil pointer dereference
. But it still hard to find the problem.
The error message is saying that is the gzip.NewReader...
. Why golang
is reporting it? How could i solve it?
goroutine 1 [running]:
github.com/hensg/pushego/aws.GetGZIPProductsDump(0x7ffead44098f, 0xc, 0x7ffead44099f, 0x2f, 0x0, 0xc420190580, 0x0, 0x0)
/home/henrique/go/src/github.com/hensg/pushego/aws/s3.go:47 +0x2e7
main.main()
/home/henrique/go/src/github.com/hensg/pushego/pushego.go:28 +0x249
Main code (pushego.go)
<!-- language: lang-go -->
package main
import (
"bufio"
"flag"
"log"
"os"
"time"
"github.com/hensg/pushego/aws"
)
func init() {
log.SetOutput(os.Stdout)
}
func main() {
log.Println("Starting...")
var bucket, key string
var timeout time.Duration
flag.StringVar(&bucket, "b", "", "Bucket name")
flag.StringVar(&key, "k", "", "Object key name")
flag.DurationVar(&timeout, "d", 0, "Download timeout")
flag.Parse()
gzipReader, err := aws.GetGZIPDump(bucket, key, timeout) // line 28
if err != nil {
log.Fatal("Failed to create GZIP reader")
}
defer gzipReader.Close()
scanner := bufio.NewScanner(gzipReader)
for scanner.Scan() {
log.Printf("Read: %s\n", scanner.Text())
}
log.Printf("Successfully download file from %s/%s\n", bucket, key)
}
Aws code (aws/s3.go)
<!-- language: lang-go -->
package aws
import (
"compress/gzip"
"context"
"log"
"time"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/awserr"
"github.com/aws/aws-sdk-go/aws/request"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/s3"
)
func GetGZIPProductsDump(bucket, key string, timeout time.Duration) (*gzip.Reader, error) {
sess := session.Must(session.NewSession())
svc := s3.New(sess)
// Create a context with a timeout that will abort the download if it takes
// more than the passed in timeout.
ctx := context.Background()
var cancelFn func()
if timeout > 0 {
ctx, cancelFn = context.WithTimeout(ctx, timeout)
}
// Ensure the context is canceled to prevent leaking.
// See context package for more information, https://golang.org/pkg/context/
defer cancelFn()
// Uploads the object to S3. The Context will interrupt the request if the
// timeout expires.
resp, err := svc.GetObjectWithContext(ctx, &s3.GetObjectInput{
Bucket: aws.String(bucket),
Key: aws.String(key),
})
if err != nil {
if aerr, ok := err.(awserr.Error); ok && aerr.Code() == request.CanceledErrorCode {
// If the SDK can determine the request or retry delay was canceled
// by a context the CanceledErrorCode error code will be returned.
log.Fatal("Download canceled due to timeout, %v\n", err)
} else {
log.Fatal("Failed to download object, %v\n", err)
}
}
return gzip.NewReader(resp.Body) // line 47
}
答案1
得分: 0
问题与defer cancelFn()
相关,当前如果超时为0,则不设置该函数,这会导致空指针引发恐慌,因此,您应该将defer
移到前面的if语句内,因为这是唯一需要使用它的地方。aws/s3.go
的代码应该如下所示。
...
// 使用超时创建一个上下文,如果下载时间超过传入的超时时间,则中止下载。
ctx := context.Background()
var cancelFn func()
if timeout > 0 {
ctx, cancelFn = context.WithTimeout(ctx, timeout)
// 确保上下文被取消以防止泄漏。
// 有关更多信息,请参阅上下文包,https://golang.org/pkg/context/
defer cancelFn()
}
...
英文:
The problem is related with the defer cancelFn()
, currently that func is not set if the timeout is 0, this provoke a panic with a null pointer, so, you should move the defer
inside of the previous if statement because it's the only point when is required to be used. The code of aws/s3.go
have to be as the following.
<!-- language: go -->
...
// Create a context with a timeout that will abort the download if it takes
// more than the passed in timeout.
ctx := context.Background()
var cancelFn func()
if timeout > 0 {
ctx, cancelFn = context.WithTimeout(ctx, timeout)
// Ensure the context is canceled to prevent leaking.
// See context package for more information, https://golang.org/pkg/context/
defer cancelFn()
}
...
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论