io.ReadFull和bytes.Buffer.ReadFrom在golang中的行为是什么?

huangapple go评论75阅读模式
英文:

can someone tell me what's the behavior of io.ReadFull and bytes.Buffer.ReadFrom in golang

问题

我在实现一个TCP客户端/服务器演示时遇到了问题。当我在服务器端使用io.ReadFull(conn, aByteArr)bytes.Buffer.ReadFrom(conn)时,发现在客户端退出之前,服务器似乎不会读取连接中的数据,换句话说,服务器被阻塞了。但是我可以使用基本的conn.Read(aBuffer)来读取数据。为什么这两种方法如此奇怪?

因为我希望我的服务器能够处理任意大小的数据,所以我不想使用基本的方式,也就是conn.Read(),这种方式必须先创建一个指定大小的字节切片。请帮助我。

以下是你提供的代码:

客户端:

package main

import (
    "net"
    "fmt"
    "bufio"
    "time"
    "runtime"
)

func send(s string, ch chan string){
    conn, err := net.Dial("tcp", ":4000")
    if err != nil {
        fmt.Println(err)
    }   
    fmt.Fprintf(conn, s)
    fmt.Println("send: ", s)                                                                                                                                                                                      
    /*  
    s := "server run"
    conn.Write([]byte(s))
    */
    status, err := bufio.NewReader(conn).ReadString('\n')
    if err != nil {
        fmt.Println("error: ", err)
    }   
    ch <- status 
}
func main(){
    runtime.GOMAXPROCS(runtime.NumCPU())
    fmt.Println("cpu: ", runtime.NumCPU())
    ch := make(chan string, 5)
    timeout := time.After(10 * time.Second)
    i := 0
    
    for{
        go send(fmt.Sprintf("%s%d", "client", i), ch) 
        i++ 
        select {
            case ret := <-ch:
                fmt.Println(ret)
            case <-timeout:
                fmt.Println("time out")
                return
        }   
    }   
}

服务器:

package main

import (
    "net"
    "log"
    "io"
    "fmt"
    "time"
    //"bytes"
)

func main(){
    // Listen on TCP port 2000 on all interfaces.
    l, err := net.Listen("tcp", ":4000")
    if err != nil {
        log.Fatal(err)
    }   
    defer l.Close()
    for {
        // Wait for a connection.
        conn, err := l.Accept()
        if err != nil {
            log.Fatal(err)
        }   
        // Handle the connection in a new goroutine.
        // The loop then returns to accepting, so that
        // multiple connections may be served concurrently.
        go func(c net.Conn) {
            fmt.Println(c.RemoteAddr())
            defer c.Close()
            // Echo all incoming data.
            /*  basic
            buf := make([]byte, 100)
            c.Read(buf)
            fmt.Println(string(buf))
            //io.Copy(c, c)
            c.Write(buf)
            // Shut down the connection.
            */

            /* use a ReadFrom
            var b bytes.Buffer                                                                                                                                                                                    
            b.ReadFrom(conn)

            fmt.Println("length: ", b.Len())
            c.Write(b.Bytes())
            */

            // use io.ReadAll
            byteArr := make([]byte, 100)
            
            n, err := io.ReadFull(c, byteArr)
            if err != nil {
                fmt.Println(err)
            }   
            fmt.Println(n, byteArr[:n], time.Now())
            n, _ = c.Write(byteArr[:n])
            fmt.Println("write: ", n, time.Now())
        }(conn)
    }
}

希望对你有所帮助!

英文:

I have a problem when I implement a tcp c/s demo, I found it's weird when I use io.ReadFull(conn, aByteArr) or bytes.Buffer.ReadFrom(conn) on the server side, it seems that the server won't read the data in the connnection until the client quit, in other words, the server is stucked, but I can use the basic conn.Read(aBuffer) to read data. why the two methods are so strange?

because I want my server to handle data of arbitrary size, so I don't like to use the basic way, I mean conn.Read(), which must make a bytes slice with specified size first. Please help me.

i can give my code:
client:

package main
import (
&quot;net&quot;
&quot;fmt&quot;
&quot;bufio&quot;
&quot;time&quot;
&quot;runtime&quot;
)
func send(s string, ch chan string){
conn, err := net.Dial(&quot;tcp&quot;, &quot;:4000&quot;)
if err != nil {
fmt.Println(err)
}   
fmt.Fprintf(conn, s)
fmt.Println(&quot;send: &quot;, s)                                                                                                                                                                                      
/*  
s := &quot;server run&quot;
conn.Write([]byte(s))
*/
status, err := bufio.NewReader(conn).ReadString(&#39;\n&#39;)
if err != nil {
fmt.Println(&quot;error: &quot;, err)
}   
ch &lt;- status 
}
func main(){
runtime.GOMAXPROCS(runtime.NumCPU())
fmt.Println(&quot;cpu: &quot;, runtime.NumCPU())
ch := make(chan string, 5)
timeout := time.After(10 * time.Second)
i := 0
for{
go send(fmt.Sprintf(&quot;%s%d&quot;, &quot;client&quot;, i), ch) 
i++ 
select {
case ret := &lt;-ch:
fmt.Println(ret)
case &lt;-timeout:
fmt.Println(&quot;time out&quot;)
return
}   
}   
}

server:

package main
import (
&quot;net&quot;
&quot;log&quot;
&quot;io&quot;
&quot;fmt&quot;
&quot;time&quot;
//&quot;bytes&quot;
)
func main(){
// Listen on TCP port 2000 on all interfaces.
l, err := net.Listen(&quot;tcp&quot;, &quot;:4000&quot;)
if err != nil {
log.Fatal(err)
}   
defer l.Close()
for {
// Wait for a connection.
conn, err := l.Accept()
if err != nil {
log.Fatal(err)
}   
// Handle the connection in a new goroutine.
// The loop then returns to accepting, so that
// multiple connections may be served concurrently.
go func(c net.Conn) {
fmt.Println(c.RemoteAddr())
defer c.Close()
// Echo all incoming data.
/*  basic
buf := make([]byte, 100)
c.Read(buf)
fmt.Println(string(buf))
//io.Copy(c, c)
c.Write(buf)
// Shut down the connection.
*/
/* use a ReadFrom
var b bytes.Buffer                                                                                                                                                                                    
b.ReadFrom(conn)
fmt.Println(&quot;length: &quot;, b.Len())
c.Write(b.Bytes())
*/
// use io.ReadAll
byteArr := make([]byte, 100)
n, err := io.ReadFull(c, byteArr)
if err != nil {
fmt.Println(err)
}   
fmt.Println(n, byteArr[:n], time.Now())
n, _ = c.Write(byteArr[:n])
fmt.Println(&quot;write: &quot;, n, time.Now())
}(conn)
}
}   

答案1

得分: 8

首先:你从未关闭客户端中建立的连接。每次调用send时,你都会拨号建立一个新的连接,但从未刷新或关闭该连接。这似乎非常奇怪,可能是唯一的问题(例如,如果某个层将你的数据缓冲到关闭或刷新)。

你似乎认为应该有一种简单的方法来“从某个连接或io.Reader中读取所有内容”。但实际上并没有这样的方法。如果你对此感到不满,那是不必要的。你想要读取“任意大小的数据”,但是任意大小可能意味着418 PB。这是非常大的数据量,可能需要一些时间。而且我敢打赌你没有足够的计算能力来处理这样的数据量。读取任意大小的数据基本上需要分块读取和处理,因为你无法处理418 PB的数据。

分块读取是io.Reader提供的功能。它有些笨拙。这就是为什么很多协议都以数据大小开始的原因:你读取6个字节,比如“ 1423”,解析整数,然后知道你的消息长度为1432字节。从那里开始,你可以使用bufio.Scannerbytes.Bufferio.ReadFull等提供的便利函数。即使使用这些函数,也需要处理EOF并可能失败。

如果你的消息不以某种长度指示开始(或者长度固定:-)),你将不得不一直读取,直到遇到EOF为止。为了让EOF到达,你必须关闭发送端,否则连接仍然打开,可能会在将来的某个时候决定发送更多的数据。

英文:

First of all: You are never closing the made connection in your client. On each invocation of send you dail a new conn but never flush or close that connection. This seems pretty strange and might be the sole issue here (e.g. if some layer buffers your stuff up to a close or a flush).

It seems you think there should be an easy way of "read all from" some connection or io.Reader.
There isn't. If you are upset about this, you shouldn't be. You want to read "data of arbitrary size" but arbitrary size can mean 418 Peta bytes. This is a lot and might take some time. And I'll bet you do not have the computing power to handle such data sizes. Reading arbitrary size basically calls for reading it in chunks, and processing it in chunks, as you just cannot handle 418 Peta bytes.

Reading in chunks is what io.Reader provides. It is clumsy. That is the reason a lot of protocols start of with the size of data: You read 6 bytes like " 1423", parse the integer number and know your message is 1432 bytes long. From there on you can use convenience functions provided by bufio.Scanner, bytes.Buffer, io.ReadFull and that like. And even those require EOFs and may fail.

If your messages do not start with some length indication (or are fixed length io.ReadFull和bytes.Buffer.ReadFrom在golang中的行为是什么? you will have to read until EOF. For this EOF to arrive you must close the sending side, otherwise the connection is still open and might decide to send more stuff sometime in the future.

huangapple
  • 本文由 发表于 2014年1月15日 21:28:11
  • 转载请务必保留本文链接:https://go.coder-hub.com/21138693.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定