Golang IO读取文件无效的内存地址

huangapple go评论182阅读模式
英文:

Golang IO reading file invalid memory address

问题

我正在尝试从AWS S3中读取一个GZIP文件时遇到一些问题。
我只有一个简单的代码,但是我得到了错误panic: runtime error: invalid memory address or nil pointer dereference。但是很难找到问题所在。

错误消息说的是gzip.NewReader...。为什么golang会报告这个错误?我该如何解决它?

主要代码(pushego.go):

package main

import (
	"bufio"
	"flag"
	"log"
	"os"
	"time"

	"github.com/hensg/pushego/aws"
)

func init() {
	log.SetOutput(os.Stdout)
}

func main() {
	log.Println("Starting...")

	var bucket, key string
	var timeout time.Duration

	flag.StringVar(&bucket, "b", "", "Bucket name")
	flag.StringVar(&key, "k", "", "Object key name")
	flag.DurationVar(&timeout, "d", 0, "Download timeout")
	flag.Parse()

	gzipReader, err := aws.GetGZIPDump(bucket, key, timeout) // line 28
	if err != nil {
		log.Fatal("Failed to create GZIP reader")
	}
	defer gzipReader.Close()

	scanner := bufio.NewScanner(gzipReader)
	for scanner.Scan() {
		log.Printf("Read: %s\n", scanner.Text())
	}

	log.Printf("Successfully download file from %s/%s\n", bucket, key)
}

AWS代码(aws/s3.go):

package aws

import (
	"compress/gzip"
	"context"
	"log"
	"time"

	"github.com/aws/aws-sdk-go/aws"
	"github.com/aws/aws-sdk-go/aws/awserr"
	"github.com/aws/aws-sdk-go/aws/request"
	"github.com/aws/aws-sdk-go/aws/session"
	"github.com/aws/aws-sdk-go/service/s3"
)

func GetGZIPProductsDump(bucket, key string, timeout time.Duration) (*gzip.Reader, error) {
	sess := session.Must(session.NewSession())
	svc := s3.New(sess)

	// Create a context with a timeout that will abort the download if it takes
	// more than the passed in timeout.
	ctx := context.Background()
	var cancelFn func()
	if timeout > 0 {
		ctx, cancelFn = context.WithTimeout(ctx, timeout)
	}
	// Ensure the context is canceled to prevent leaking.
	// See context package for more information, https://golang.org/pkg/context/
	defer cancelFn()

	// Uploads the object to S3. The Context will interrupt the request if the
	// timeout expires.
	resp, err := svc.GetObjectWithContext(ctx, &s3.GetObjectInput{
		Bucket: aws.String(bucket),
		Key:    aws.String(key),
	})
	if err != nil {
		if aerr, ok := err.(awserr.Error); ok && aerr.Code() == request.CanceledErrorCode {
			// If the SDK can determine the request or retry delay was canceled
			// by a context the CanceledErrorCode error code will be returned.
			log.Fatal("Download canceled due to timeout, %v\n", err)
		} else {
			log.Fatal("Failed to download object, %v\n", err)
		}
	}

	return gzip.NewReader(resp.Body) // line 47
}

希望这可以帮助你解决问题。

英文:

I'm having some troubles trying to read a GZIP file from AWS S3.
I have only a simple code and I'm getting the error panic: runtime error: invalid memory address or nil pointer dereference. But it still hard to find the problem.

The error message is saying that is the gzip.NewReader.... Why golang is reporting it? How could i solve it?

goroutine 1 [running]:
github.com/hensg/pushego/aws.GetGZIPProductsDump(0x7ffead44098f, 0xc, 0x7ffead44099f, 0x2f, 0x0, 0xc420190580, 0x0, 0x0)
        /home/henrique/go/src/github.com/hensg/pushego/aws/s3.go:47 +0x2e7
main.main()
        /home/henrique/go/src/github.com/hensg/pushego/pushego.go:28 +0x249

Main code (pushego.go)
<!-- language: lang-go -->

package main

import (
	&quot;bufio&quot;
	&quot;flag&quot;
	&quot;log&quot;
	&quot;os&quot;
	&quot;time&quot;

	&quot;github.com/hensg/pushego/aws&quot;
)

func init() {
	log.SetOutput(os.Stdout)
}

func main() {
	log.Println(&quot;Starting...&quot;)

	var bucket, key string
	var timeout time.Duration

	flag.StringVar(&amp;bucket, &quot;b&quot;, &quot;&quot;, &quot;Bucket name&quot;)
	flag.StringVar(&amp;key, &quot;k&quot;, &quot;&quot;, &quot;Object key name&quot;)
	flag.DurationVar(&amp;timeout, &quot;d&quot;, 0, &quot;Download timeout&quot;)
	flag.Parse()

	gzipReader, err := aws.GetGZIPDump(bucket, key, timeout) // line 28
	if err != nil {
		log.Fatal(&quot;Failed to create GZIP reader&quot;)
	}
   defer gzipReader.Close()

	scanner := bufio.NewScanner(gzipReader)
	for scanner.Scan() {
		log.Printf(&quot;Read: %s\n&quot;, scanner.Text())
	}

	log.Printf(&quot;Successfully download file from %s/%s\n&quot;, bucket, key)
}

Aws code (aws/s3.go)
<!-- language: lang-go -->

package aws

import (
	&quot;compress/gzip&quot;
	&quot;context&quot;
	&quot;log&quot;
	&quot;time&quot;

	&quot;github.com/aws/aws-sdk-go/aws&quot;
	&quot;github.com/aws/aws-sdk-go/aws/awserr&quot;
	&quot;github.com/aws/aws-sdk-go/aws/request&quot;
	&quot;github.com/aws/aws-sdk-go/aws/session&quot;
	&quot;github.com/aws/aws-sdk-go/service/s3&quot;
)

func GetGZIPProductsDump(bucket, key string, timeout time.Duration) (*gzip.Reader, error) {
	sess := session.Must(session.NewSession())
	svc := s3.New(sess)

	// Create a context with a timeout that will abort the download if it takes
	// more than the passed in timeout.
	ctx := context.Background()
	var cancelFn func()
	if timeout &gt; 0 {
		ctx, cancelFn = context.WithTimeout(ctx, timeout)
	}
	// Ensure the context is canceled to prevent leaking.
	// See context package for more information, https://golang.org/pkg/context/
	defer cancelFn()

	// Uploads the object to S3. The Context will interrupt the request if the
	// timeout expires.
	resp, err := svc.GetObjectWithContext(ctx, &amp;s3.GetObjectInput{
		Bucket: aws.String(bucket),
		Key:    aws.String(key),
	})
	if err != nil {
		if aerr, ok := err.(awserr.Error); ok &amp;&amp; aerr.Code() == request.CanceledErrorCode {
			// If the SDK can determine the request or retry delay was canceled
			// by a context the CanceledErrorCode error code will be returned.
			log.Fatal(&quot;Download canceled due to timeout, %v\n&quot;, err)
		} else {
			log.Fatal(&quot;Failed to download object, %v\n&quot;, err)
		}
	}

	return gzip.NewReader(resp.Body) // line 47
}

答案1

得分: 0

问题与defer cancelFn()相关,当前如果超时为0,则不设置该函数,这会导致空指针引发恐慌,因此,您应该将defer移到前面的if语句内,因为这是唯一需要使用它的地方。aws/s3.go的代码应该如下所示。

...
// 使用超时创建一个上下文,如果下载时间超过传入的超时时间,则中止下载。
ctx := context.Background()
var cancelFn func()
if timeout > 0 {
    ctx, cancelFn = context.WithTimeout(ctx, timeout)
    // 确保上下文被取消以防止泄漏。
    // 有关更多信息,请参阅上下文包,https://golang.org/pkg/context/
    defer cancelFn()
}
...
英文:

The problem is related with the defer cancelFn(), currently that func is not set if the timeout is 0, this provoke a panic with a null pointer, so, you should move the defer inside of the previous if statement because it's the only point when is required to be used. The code of aws/s3.go have to be as the following.

<!-- language: go -->

...
// Create a context with a timeout that will abort the download if it takes
// more than the passed in timeout.
ctx := context.Background()
var cancelFn func()
if timeout &gt; 0 {
    ctx, cancelFn = context.WithTimeout(ctx, timeout)
    // Ensure the context is canceled to prevent leaking.
    // See context package for more information, https://golang.org/pkg/context/
    defer cancelFn()
}
...

huangapple
  • 本文由 发表于 2017年4月24日 23:25:16
  • 转载请务必保留本文链接:https://go.coder-hub.com/43592034.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定