英文:
go-getter and context deadline exceeded
问题
我想使用go-getter v2从HTTP下载一些巨大的ZIP文件并将它们解压缩。
我尝试了以下代码（注意：defaultProgressBar是从这里获取的）：
package main
import (
"context"
"log"
getter "github.com/hashicorp/go-getter/v2"
)
func main() {
ctx := context.Background()
req := &getter.Request{
Src: "https://huge-file.zip",
Dst: "data/",
Pwd: "./",
GetMode: getter.ModeAny,
ProgressListener: defaultProgressBar,
}
// client := getter.DefaultClient
client := getter.Client{}
_, err := client.Get(ctx, req)
if err != nil {
log.Fatal(err)
}
}
这段代码总是返回context deadline exceeded。我应该了解关于上下文使用的内容吗？
英文:
I would like to use go-getter v2 to download from http some huge zip files and have them unpacked.
I try this code (note: defaultProgressBar taken from here)
package main
import (
"context"
"log"
getter "github.com/hashicorp/go-getter/v2"
)
func main() {
ctx := context.Background()
req := &getter.Request{
Src: "https://huge-file.zip",
Dst: "data/",
Pwd: "./",
GetMode: getter.ModeAny,
ProgressListener: defaultProgressBar,
}
// client := getter.DefaultClient
client := getter.Client{}
_, err := client.Get(ctx, req)
if err != nil {
log.Fatal(err)
}
}
This code always returns context deadline exceeded. What should i known about context usage?
答案1
得分: 0
这是关于getter.Client的文档：
type Client struct {
// Getters 是此客户端支持的协议列表。如果为nil，则使用默认的 Getters 变量。
Getters []Getter
// 其他字段被省略。
}
默认的 Getters 中的 HttpGetter 配置了一个 30秒 的 ReadTimeout（参见 源代码）：
httpGetter := &HttpGetter{
Netrc: true,
XTerraformGetDisabled: true,
HeadFirstTimeout: 10 * time.Second,
ReadTimeout: 30 * time.Second,
}
因此，错误是由默认的 HttpGetter 报告的。如果下载大文件需要超过30秒的时间，你应该使用合理的值配置 HttpGetter。请参考下面的示例：
package main
import (
"context"
"log"
"net/http"
"net/http/httptest"
"time"
getter "github.com/hashicorp/go-getter/v2"
)
func main() {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Write([]byte{'a'})
time.Sleep(31 * time.Second)
w.Write([]byte{'b'})
}))
defer ts.Close()
ctx := context.Background()
req := &getter.Request{
Src: ts.URL,
Dst: "data.txt",
Pwd: "./",
GetMode: getter.ModeAny,
}
client := &getter.Client{
Getters: []getter.Getter{
&getter.HttpGetter{
Netrc: true,
XTerraformGetDisabled: true,
HeadFirstTimeout: 10 * time.Second,
// 将 ReadTimeout 设置为大文件的合理值。
ReadTimeout: 60 * time.Second,
},
},
}
log.Println("开始下载")
_, err := client.Get(ctx, req)
if err != nil {
log.Fatal(err)
}
log.Println("完成！")
}
英文:
Here is the doc for getter.Client:
> go
> type Client struct {
> // Getters is the list of protocols supported by this client. If this
> // is nil, then the default Getters variable will be used.
> Getters []Getter
>
> // other fields are omitted.
> }
>
And the HttpGetter in the default Getters is configured with a ReadTimeout of 30s (see the source code):
httpGetter := &HttpGetter{
Netrc: true,
XTerraformGetDisabled: true,
HeadFirstTimeout: 10 * time.Second,
ReadTimeout: 30 * time.Second,
}
So the error is reported by the default HttpGetter. If the huge file takes more than 30s to download, you should configure the HttpGetter with a reasonable value. See the demo below:
package main
import (
"context"
"log"
"net/http"
"net/http/httptest"
"time"
getter "github.com/hashicorp/go-getter/v2"
)
func main() {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Write([]byte{'a'})
time.Sleep(31 * time.Second)
w.Write([]byte{'b'})
}))
defer ts.Close()
ctx := context.Background()
req := &getter.Request{
Src: ts.URL,
Dst: "data.txt",
Pwd: "./",
GetMode: getter.ModeAny,
}
client := &getter.Client{
Getters: []getter.Getter{
&getter.HttpGetter{
Netrc: true,
XTerraformGetDisabled: true,
HeadFirstTimeout: 10 * time.Second,
// Set the ReadTimeout to a reasonable value for the huge file.
ReadTimeout: 60 * time.Second,
},
},
}
log.Println("begin downloading")
_, err := client.Get(ctx, req)
if err != nil {
log.Fatal(err)
}
log.Println("done!")
}
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论