go rpc, http or websockets,which is fastest for transferring many small pieces of data, repeatedly, from one server to another

huangapple go评论90阅读模式
英文:

go rpc, http or websockets,which is fastest for transferring many small pieces of data, repeatedly, from one server to another

问题

背景:

我正在尝试在Go语言中创建一个内存和CPU分析器,并希望将信息快速传输到一个服务器上,可能是每秒钟一次。服务器将通过将数据保存到数据库和/或通过HTTP提供给网站来完成所有繁重的工作,这将减轻被分析程序的负载,以获得更准确的测量结果。传输的数据将是小块数据。我知道已经有一些库可用,但正如我所说,我正在进行实验。

传输内容类型:

我还没有确定具体的传输类型,但看起来HTTP或Websockets可以使用JSON,而RPC可以使用结构体(如果我研究得正确)。

总结:

我可能会尝试每种方法,以便亲自了解它们,但我对使用RPC和Websockets的经验有限,希望能听到一些关于哪种方法可能更快或更适合我所尝试做的事情的意见或建议:

  • HTTP
  • RPC
  • Websockets
  • 其他我没有考虑到的方法
英文:

Background

I'm experimenting creating a memory + cpu profiler in go, and wish to transfer the information quickly, maybe every second, from the program/service being profiled to a server which will do all of the heavy lifting by saving the data to a database and/or serving it via http to a site; this will reduce the load on the program being profiled for more accurate measurements. It will be small pieces of data being transferred. I know there are some libraries out there already, but like I said, experimenting.

Transfer Content Type

I have not decided on a concrete transfer type but looks like JSON for HTTP or Websockets and just the Struct for RPC (if I've done my research correctly)

Summary

I will likely try each just to see for myself, but have little experience using RPC and Websockets and would like some opinions or recommendations on which may be faster or more suitable for what I'm trying to do:

  • HTTP
  • RPC
  • Websockets
  • Anything else I'm not thinking about

答案1

得分: 2

如您在评论中提到的,HTTP并不是必需的。

在这种情况下,为了寻找最快的传输解决方案,我会完全放弃HTTP传输层,而是只使用普通的(TCP)套接字连接,因为HTTP在传输几个字节时会带来相当大的开销。

您可以定义自己的协议(可能非常简单),打开到服务器的TCP连接,并根据您的要求每秒发送数据包。

发送(和接收)数据的协议可以非常简单,例如:

  • 进行可选的身份验证或客户端/服务器识别(以确保您连接到了所需的服务器/程序)。
  • 使用标准库中的encoding/gob包以二进制形式通过连接发送数据。

因此,基本上,被分析的程序(客户端)应该打开TCP连接,并使用将连接包装在内的gob.NewEncoder()来发送数据。服务器应该接受传入的TCP连接,并使用将连接包装在内的gob.NewDecoder()来接收数据。

客户端调用Encoder.Encode()来发送分析信息,通常可以是一个结构体值。服务器调用Decoder.Decode()来接收分析信息,即客户端发送的结构体。就是这样。

使用encoding/gob包以二进制形式发送数据需要在两端使用相同的类型来描述分析数据。如果您需要更灵活性,您还可以使用encoding/json包将分析信息作为JSON文本发送/接收。不过,缺点是JSON需要发送更多的数据,并且生成和解析JSON文本所需的时间比二进制表示更长。

如果丢失一些分析数据包(或接收到重复数据包)不是问题,您可以尝试使用UDP而不是TCP,这可能更高效。

英文:

As you mentioned in your comment, HTTP is not a requirement.

In this case in search for the fastest transferring solution I would completely drop the HTTP transport layer and would use just plain (TCP) socket connections as HTTP gives quite a big overhead just for transferring a few bytes.

Define your own protocol (which may be very simple), open a TCP connection to the server, and send the data packets every seconds or so as your requirements dictate.

Your protocol for sending (and receiving) data can be as simple as:

  • Do an optional authenticating or client/server identification (to ensure you connected to the server/program you wanted to).
  • Use the encoding/gob packgae from the standard library to send data in binary form over the connection.

So basically the profiled program (client) should open the TCP connection, and use gob.NewEncoder() wrapping the connection to send data. The server should accept the incoming TCP connection and use gob.NewDecoder() to wrapping the connection to recieve data.

Client calls Encoder.Encode() so send profiling info, it can be typically a struct value. Server calls Decoder.Decode() to receive a profiling info, the struct that the client sent. That's all.

Sending data in binary form using the encoding/gob package requires you to use the same type to describe the profiling data on both sides. If you want more flexibility, you may also use the encoding/json package to send/receive profiling info as JSON text. The downside is that JSON will require more data to be sent and it takes more time to produce and parse the JSON text compared to the binary representation.

If loosing some profiling packets (or recieving duplicates) is not an issue, you may want to try/experiment using UDP instead of TCP which may be even more efficient.

huangapple
  • 本文由 发表于 2015年9月14日 09:29:13
  • 转载请务必保留本文链接:https://go.coder-hub.com/32556174.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定