英文:
Is there a way to directly copy golang strings to a pre-allocated C char buffer
问题
我有一个变量(但在组合大小上有限制)的GoStrings需要传递给C,我希望尽可能地廉价地完成这个操作。我将多次执行此操作(因此可以将重用的预分配缓冲区视为零成本)。
我的初始方法是循环遍历GoStrings,将每个字符串转换为CString并将其推送到C。
for _, str := range mystrings {
cstr := C.CString(str)
defer C.free(unsafe.Pointer(cstr))
C.push_str(pushStrFn, cstr)
}
当然,这要归功于C.CString
,这将执行N个堆分配,以及N个CGo调用,这些都不是廉价的。
接下来是在Go中使用一个在一开始分配的strings.Builder
构建一个大字符串,然后在一个CGo调用中传递它以及一些长度信息。这是一个CString调用和一个CGo调用,这是一个重大的改进。
builder.Reset()
for _, str := range mystrings {
builder.WriteString(str)
}
C.push_strs(pushStrsFn, C.CString(builder.String()))
但是这种方法仍然执行了不必要的复制!理想情况下,我想预先分配一大块内存,可以将其直接传递给C,而不使用一个大的GoString中间件来复制字符串。
我可以预先分配一个大数组,并在GoStrings中迭代字符,逐个将它们复制过去。这样可以避免中间复制,但比专用的字符串复制函数(如builder的函数)慢得多。
cCharArray := C.malloc(C.size_t(MAX_SIZE) * C.size_t(unsafe.Sizeof(uintptr(0))))
goCharArray := (*[1<<30 - 1]C.char)(cCharArray)
for _, str := range mystrings {
for i, c := range str {
goCharArray[offset+i] = C.char(c)
}
}
C.push_charArray(pushCharArrayFn, (*C.char)(cCharArray))
有没有更快的方法可以做到这一点?我是否可以将C缓冲区提供给strings.Builder
,或者直接使用字符串复制函数复制到C缓冲区?
英文:
I have a variable (but capped in combined size) number of GoStrings that need to be passed to C, and I want to do it as cheaply as possible. I'm going to perform this operation many times (so pre-allocating buffers that are reused can be considered zero cost).
My initial approach was to loop over the GoStrings, converting each to a CString and pushing it to C.
for _, str := range mystrings {
cstr := C.CString(str)
defer C.free(unsafe.Pointer(cstr))
C.push_str(pushStrFn, cstr)
}
Of course, this is performing N heap allocations thanks to C.CString
, as well as N CGo invocations - all of which are not cheap.
Next up was to build a single big string in Go using an allocated-at-the-beginning-of-time strings.Builder
, and then pass it to C with some length information in a single CGo call. This is one CString invocation, and one CGo call - a substantial improvement.
builder.Reset()
for _, str := range mystrings {
builder.WriteString(str)
}
C.push_strs(pushStrsFn, C.CString(builder.String()))
But this approach is still performing an unnecessary copy! Ideally I'd like to pre-allocate a big chunk of memory that I can pass to C, and just copy the strings directly to it without using a big GoString intermediary.
I'm able to pre-allocate a big array ahead of time, and iterate over the characters in the GoStrings, copying them over one at a time. This avoids the intermediate copy, but is substantially slower than a dedicated string-copying function (like that of the builder).
cCharArray := C.malloc(C.size_t(MAX_SIZE) * C.size_t(unsafe.Sizeof(uintptr(0))))
goCharArray := (*[1<<30 - 1]C.char)(cCharArray)
for _, str := range mystrings {
for i, c := range str {
goCharArray[offset+i] = C.char(c)
}
}
C.push_charArray(pushCharArrayFn, (*C.char)(cCharArray))
Is there a faster way to do this that I'm missing? Can I somehow feed a C buffer to strings.Builder
, or use a string-copying function directly to the C buffer?
答案1
得分: 1
你试过使用C.strncpy
函数或者它的包装器吗?你可以参考这个链接:https://github.com/chai2010/cgo/blob/master/char.go#L61。如果速度还不够快,你可以尝试使用unsafe
将Go字符串的指针转换为CString
指针。
英文:
Did you try out C.strncpy
, or this wrapper over it ? https://github.com/chai2010/cgo/blob/master/char.go#L61 . If this is not faster you could try to cast the pointer to the Go string into a CString pointer using unsafe.
答案2
得分: 0
使用unsafe包将C分配的缓冲区创建为字节切片。
然后,您可以直接将数据复制到该切片中,或者将其作为长度为零的切片附加到bytes.Buffer中并使用WriteString方法。但是,您必须小心不要超出原始容量,否则Buffer将会将数据复制到新的后备缓冲区中。
英文:
Use the unsafe package to create a byte slice out of a C allocated buffer.
You can then copy into that slice directly or stick as zero length reslice of it into a bytes.Buffer and use WriteString. You have to be careful not to exceed your original capacity or the Buffer will copy your data to a new backing buffer.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论