有没有一种方法可以直接将 Golang 字符串复制到预分配的 C 字符缓冲区中?

huangapple go评论91阅读模式
英文:

Is there a way to directly copy golang strings to a pre-allocated C char buffer

问题

我有一个变量(但在组合大小上有限制)的GoStrings需要传递给C,我希望尽可能地廉价地完成这个操作。我将多次执行此操作(因此可以将重用的预分配缓冲区视为零成本)。

我的初始方法是循环遍历GoStrings,将每个字符串转换为CString并将其推送到C。

for _, str := range mystrings {
   cstr := C.CString(str)
   defer C.free(unsafe.Pointer(cstr))
   C.push_str(pushStrFn, cstr)
}

当然,这要归功于C.CString,这将执行N个堆分配,以及N个CGo调用,这些都不是廉价的。

接下来是在Go中使用一个在一开始分配的strings.Builder构建一个大字符串,然后在一个CGo调用中传递它以及一些长度信息。这是一个CString调用和一个CGo调用,这是一个重大的改进。

builder.Reset()
for _, str := range mystrings {
   builder.WriteString(str)
}
C.push_strs(pushStrsFn, C.CString(builder.String()))

但是这种方法仍然执行了不必要的复制!理想情况下,我想预先分配一大块内存,可以将其直接传递给C,而不使用一个大的GoString中间件来复制字符串。

我可以预先分配一个大数组,并在GoStrings中迭代字符,逐个将它们复制过去。这样可以避免中间复制,但比专用的字符串复制函数(如builder的函数)慢得多。

cCharArray := C.malloc(C.size_t(MAX_SIZE) * C.size_t(unsafe.Sizeof(uintptr(0))))
goCharArray := (*[1<<30 - 1]C.char)(cCharArray)
for _, str := range mystrings {
   for i, c := range str {
      goCharArray[offset+i] = C.char(c)
   }
}
C.push_charArray(pushCharArrayFn, (*C.char)(cCharArray))

有没有更快的方法可以做到这一点?我是否可以将C缓冲区提供给strings.Builder,或者直接使用字符串复制函数复制到C缓冲区?

英文:

I have a variable (but capped in combined size) number of GoStrings that need to be passed to C, and I want to do it as cheaply as possible. I'm going to perform this operation many times (so pre-allocating buffers that are reused can be considered zero cost).

My initial approach was to loop over the GoStrings, converting each to a CString and pushing it to C.

for _, str := range mystrings {
   cstr := C.CString(str)
   defer C.free(unsafe.Pointer(cstr))
   C.push_str(pushStrFn, cstr)
}

Of course, this is performing N heap allocations thanks to C.CString, as well as N CGo invocations - all of which are not cheap.

Next up was to build a single big string in Go using an allocated-at-the-beginning-of-time strings.Builder, and then pass it to C with some length information in a single CGo call. This is one CString invocation, and one CGo call - a substantial improvement.

builder.Reset()
for _, str := range mystrings {
   builder.WriteString(str)
}
C.push_strs(pushStrsFn, C.CString(builder.String()))

But this approach is still performing an unnecessary copy! Ideally I'd like to pre-allocate a big chunk of memory that I can pass to C, and just copy the strings directly to it without using a big GoString intermediary.

I'm able to pre-allocate a big array ahead of time, and iterate over the characters in the GoStrings, copying them over one at a time. This avoids the intermediate copy, but is substantially slower than a dedicated string-copying function (like that of the builder).

cCharArray := C.malloc(C.size_t(MAX_SIZE) * C.size_t(unsafe.Sizeof(uintptr(0))))
goCharArray := (*[1&lt;&lt;30 - 1]C.char)(cCharArray)
for _, str := range mystrings {
   for i, c := range str {
      goCharArray[offset+i] = C.char(c)
   }
}
C.push_charArray(pushCharArrayFn, (*C.char)(cCharArray))

Is there a faster way to do this that I'm missing? Can I somehow feed a C buffer to strings.Builder, or use a string-copying function directly to the C buffer?

答案1

得分: 1

你试过使用C.strncpy函数或者它的包装器吗?你可以参考这个链接:https://github.com/chai2010/cgo/blob/master/char.go#L61。如果速度还不够快,你可以尝试使用unsafe将Go字符串的指针转换为CString指针。

英文:

Did you try out C.strncpy, or this wrapper over it ? https://github.com/chai2010/cgo/blob/master/char.go#L61 . If this is not faster you could try to cast the pointer to the Go string into a CString pointer using unsafe.

答案2

得分: 0

使用unsafe包将C分配的缓冲区创建为字节切片。

然后,您可以直接将数据复制到该切片中,或者将其作为长度为零的切片附加到bytes.Buffer中并使用WriteString方法。但是,您必须小心不要超出原始容量,否则Buffer将会将数据复制到新的后备缓冲区中。

英文:

Use the unsafe package to create a byte slice out of a C allocated buffer.

You can then copy into that slice directly or stick as zero length reslice of it into a bytes.Buffer and use WriteString. You have to be careful not to exceed your original capacity or the Buffer will copy your data to a new backing buffer.

huangapple
  • 本文由 发表于 2021年10月2日 02:22:25
  • 转载请务必保留本文链接:https://go.coder-hub.com/69410435.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定