英文:
Does Go cache DNS lookups?
问题
我正在构建一个测试爬虫，并想知道Go（golang）是否缓存DNS查询。我在dnsclient中没有看到任何关于缓存的内容。这似乎是在任何爬虫中都需要添加的重要功能，以防止大量额外的DNS查询。
Go（1.4+）是否缓存DNS查找？
如果不是的话，Debian/Ubuntu/Linux、Windows或Darwin/OSX是否在网络层面上进行任何缓存，从而使Go受益？
英文:
I am building a test crawler and wanted to know if Go (golang) caches DNS queries. I don't see anything about caching in the dnsclient. This seems like an important thing to add to any crawler to prevent lots of extra DNS queries.
Does Go (1.4+) cache DNS lookups?
If not, does debian/ubuntu/linux, windows, or darwin/OSX do any caching at the network level Go benefits from?
答案1
得分: 21
你的问题的答案是否定的。标准库解析器中没有内置的 DNS 缓存。它会有帮助吗？在某些情况下可能有帮助。我们的组织在每个服务器上运行一个本地 DNS 缓存，并将 resolv.conf 指向那里。所以在语言中有缓存可能对我们帮助不大。
有一些解决方案可以帮助你。这个包似乎有一个相当不错的解决方案。根据他们自述文件中的片段，你甚至可以这样做：
http.DefaultClient.Transport = &http.Transport {
MaxIdleConnsPerHost: 64,
Dial: func(network string, address string) (net.Conn, error) {
separator := strings.LastIndex(address, ":")
ip, _ := dnscache.FetchString(address[:separator])
return net.Dial("tcp", ip + address[separator:])
},
}
这样可以为所有从 http.Get 和相关函数发起的 HTTP 请求启用它。
英文:
The answer to your question is no. There is no built-in dns caching in the std lib resolver. Would it be helpful? Maybe in some cases. Our org runs a local dns cache on each server and points resolv.conf there. So it wouldn't necessarily help us much to have caching in the language.
There are some solutions that could help you. This package seems to have a pretty good solution. From the snippet in their readme you could even do:
http.DefaultClient.Transport = &http.Transport {
MaxIdleConnsPerHost: 64,
Dial: func(network string, address string) (net.Conn, error) {
separator := strings.LastIndex(address, ":")
ip, _ := dnscache.FetchString(address[:separator])
return net.Dial("tcp", ip + address[separator:])
},
}
To enable it for all http requests from http.Get and friends.
答案2
得分: 5
Go解析器不进行任何进程内缓存。虽然你可以自己编写一个缓存，但最好的选择可能是在每台机器上运行一个系统范围的DNS缓存。（我最喜欢的是dnsmasq）。
英文:
The Go resolver does not do any in-process caching. While it would be possible to roll your own, your best bet is probably to run a system-wide DNS cache on each machine. (My favourite being dnsmasq.)
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论