Go SQlite 并发问题

huangapple go评论91阅读模式
英文:

Go SQlite Concurrency Issue

问题

我一直在使用这个Go语言的SQLite库进行一些测试,并且在进行一些压力测试时,遇到了一些奇怪的错误。

我试图运行这段代码:

package main

import "code.google.com/p/go-sqlite/go1/sqlite3"
import "fmt"
import "sync"

func main() {
    var wg sync.WaitGroup
    wg.Add(100)
    for i := 0; i < 100; i++ {
        query(&wg, i)
    }
    wg.Wait()
}

func query(wg *sync.WaitGroup, id int) {
    defer wg.Done()
    c, _ := sqlite3.Open("sqlite.db")
    for i := 0; i < 100; i++{
        args := sqlite3.NamedArgs{"$bval": "demo"}
        sql := "SELECT rowid, * FROM x WHERE b = $bval"
        row := make(sqlite3.RowMap)
        for s, err := c.Query(sql, args); err == nil; err = s.Next() {
                var rowid int64
                s.Scan(&rowid, row)     // Assigns 1st column to rowid, the rest to row
                fmt.Println(rowid, row) // Prints "1 map[a:1 b:demo c:<nil>]"
        }
    }
    c.Close()
    fmt.Println("Worker ", id, " is done")
}

默认情况下,没有问题;但是当我将GOMAXPROCS增加到大于1时,程序会在任意点崩溃,并显示以下错误:

fatal error: unexpected signal during runtime execution
[signal 0xb code=0x1 addr=0xc0000000d pc=0x4100b9]

runtime stack:
runtime.gothrow(0x5a1610, 0x2a)
    /usr/lib/go/src/runtime/panic.go:503 +0x8e fp=0x7f4bd5ffa830 sp=0x7f4bd5ffa818
runtime.sigpanic()
    /usr/lib/go/src/runtime/sigpanic_unix.go:14 +0x5e fp=0x7f4bd5ffa880 sp=0x7f4bd5ffa830

goroutine 1 [syscall, locked to thread]:
runtime.cgocall_errno(0x408043, 0xc20808fcf8, 0x0)
    /usr/lib/go/src/runtime/cgocall.go:130 +0xf5 fp=0xc20808fcd8 sp=0xc20808fcb0
code.google.com/p/go-sqlite/go1/sqlite3._Cfunc_sqlite3_prepare_v2(0x7f4bc00023b8, 0xc2080483f0, 0xffffffff, 0xc20808fd30, 0xc20808fd28, 0x0)
    /home/eumen/Programming/gocode/src/code.google.com/p/go-sqlite/go1/sqlite3/:534 +0x43 fp=0xc20808fcf8 sp=0xc20808fcd8
code.google.com/p/go-sqlite/go1/sqlite3.newStmt(0xc2080482d0, 0x599e90, 0x26, 0xc20804b8c8, 0x0, 0x0)
    /home/eumen/Programming/gocode/src/code.google.com/p/go-sqlite/go1/sqlite3/sqlite3.go:583 +0xe4 fp=0xc20808fdb0 sp=0xc20808fcf8
code.google.com/p/go-sqlite/go1/sqlite3.(*Conn).Query(0xc2080482d0, 0x599e90, 0x26, 0xc20808fee0, 0x1, 0x1, 0xc20807a000, 0x0, 0x0)
    /home/eumen/Programming/gocode/src/code.google.com/p/go-sqlite/go1/sqlite3/sqlite3.go:276 +0xc3 fp=0xc20808fe00 sp=0xc20808fdb0
main.query(0xc208022020, 0x16)
    /home/eumen/Programming/Go/practice/sqlite-sync.go:26 +0x261 fp=0xc20808ff70 sp=0xc20808fe00
main.main()
    /home/eumen/Programming/Go/practice/sqlite-sync.go:11 +0x66 fp=0xc20808ff98 sp=0xc20808ff70
runtime.main()
    /usr/lib/go/src/runtime/proc.go:63 +0xf3 fp=0xc20808ffe0 sp=0xc20808ff98
runtime.goexit()
    /usr/lib/go/src/runtime/asm_amd64.s:2232 +0x1 fp=0xc20808ffe8 sp=0xc20808ffe0

goroutine 2 [force gc (idle)]:
runtime.gopark(0x496870, 0x831d30, 0x583f70, 0xf)
    /usr/lib/go/src/runtime/proc.go:130 +0x105 fp=0xc20801e798 sp=0xc20801e768
runtime.goparkunlock(0x831d30, 0x583f70, 0xf)
    /usr/lib/go/src/runtime/proc.go:136 +0x48 fp=0xc20801e7c0 sp=0xc20801e798
runtime.forcegchelper()
    /usr/lib/go/src/runtime/proc.go:99 +0xce fp=0xc20801e7e0 sp=0xc20801e7c0
runtime.goexit()
    /usr/lib/go/src/runtime/asm_amd64.s:2232 +0x1 fp=0xc20801e7e8 sp=0xc20801e7e0
created by runtime.init·4
    /usr/lib/go/src/runtime/proc.go:87 +0x25

goroutine 3 [GC sweep wait]:
runtime.gopark(0x496870, 0x838ff8, 0x580b50, 0xd)
    /usr/lib/go/src/runtime/proc.go:130 +0x105 fp=0xc208021f98 sp=0xc208021f68
runtime.goparkunlock(0x838ff8, 0x580b50, 0xd)
    /usr/lib/go/src/runtime/proc.go:136 +0x48 fp=0xc208021fc0 sp=0xc208021f98
runtime.bgsweep()
    /usr/lib/go/src/runtime/mgc0.go:98 +0xbc fp=0xc208021fe0 sp=0xc208021fc0
runtime.goexit()
    /usr/lib/go/src/runtime/asm_amd64.s:2232 +0x1 fp=0xc208021fe8 sp=0xc208021fe0
created by gc
    /usr/lib/go/src/runtime/mgc0.c:1386

goroutine 4 [finalizer wait]:
runtime.gopark(0x496870, 0x838ff0, 0x583ad0, 0xe)
    /usr/lib/go/src/runtime/proc.go:130 +0x105 fp=0xc20801d730 sp=0xc20801d700
runtime.goparkunlock(0x838ff0, 0x583ad0, 0xe)
    /usr/lib/go/src/runtime/proc.go:136 +0x48 fp=0xc20801d758 sp=0xc20801d730
runtime.runfinq()
    /usr/lib/go/src/runtime/malloc.go:727 +0xba fp=0xc20801d7e0 sp=0xc20801d758
runtime.goexit()
    /usr/lib/go/src/runtime/asm_amd64.s:2232 +0x1 fp=0xc20801d7e8 sp=0xc20801d7e0
created by runtime.createfing
    /usr/lib/go/src/runtime/malloc.go:707 +0x5e

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
    /usr/lib/go/src/runtime/asm_amd64.s:2232 +0x1 fp=0xc208032fe8 sp=0xc208032fe0
signal: aborted (core dumped)

只有当我增加GOMAXPROCS时才会出现此错误,如果我使用go query(&wg, i)也会出现此错误。奇怪的是,如果不使用循环,或者将i设置为足够小的数字,错误也不会发生。有人能解释一下这里发生了什么吗?

英文:

I've been playing around with this SQLite library for Go, and in doing some stress testing, have come across some odd errors.

I'm attempting to run this code:

package main

import &quot;code.google.com/p/go-sqlite/go1/sqlite3&quot;
import &quot;fmt&quot;
import &quot;sync&quot;

func main() {
    var wg sync.WaitGroup
    wg.Add(100)
    for i := 0; i &lt; 100; i++ {
        query(&amp;wg, i)
    }
    wg.Wait()
}

func query(wg *sync.WaitGroup, id int) {
    defer wg.Done()
    c, _ := sqlite3.Open(&quot;sqlite.db&quot;)
    for i := 0; i &lt; 100; i++{
        args := sqlite3.NamedArgs{&quot;$bval&quot;: &quot;demo&quot;}
        sql := &quot;SELECT rowid, * FROM x WHERE b = $bval&quot;
        row := make(sqlite3.RowMap)
        for s, err := c.Query(sql, args); err == nil; err = s.Next() {
                var rowid int64
                s.Scan(&amp;rowid, row)     // Assigns 1st column to rowid, the rest to row
                fmt.Println(rowid, row) // Prints &quot;1 map[a:1 b:demo c:&lt;nil&gt;]&quot;
        }
    }
    c.Close()
    fmt.Println(&quot;Worker &quot;, id, &quot; is done&quot;)
}

By default, there are no issues; But when I increase GOMAXPROCS to more than 1, the program crashes at arbitrary points, giving the error:

fatal error: unexpected signal during runtime execution
[signal 0xb code=0x1 addr=0xc0000000d pc=0x4100b9]

runtime stack:
runtime.gothrow(0x5a1610, 0x2a)
    /usr/lib/go/src/runtime/panic.go:503 +0x8e fp=0x7f4bd5ffa830 sp=0x7f4bd5ffa818
runtime.sigpanic()
    /usr/lib/go/src/runtime/sigpanic_unix.go:14 +0x5e fp=0x7f4bd5ffa880 sp=0x7f4bd5ffa830

goroutine 1 [syscall, locked to thread]:
runtime.cgocall_errno(0x408043, 0xc20808fcf8, 0x0)
    /usr/lib/go/src/runtime/cgocall.go:130 +0xf5 fp=0xc20808fcd8 sp=0xc20808fcb0
code.google.com/p/go-sqlite/go1/sqlite3._Cfunc_sqlite3_prepare_v2(0x7f4bc00023b8, 0xc2080483f0, 0xffffffff, 0xc20808fd30, 0xc20808fd28, 0x0)
    /home/eumen/Programming/gocode/src/code.google.com/p/go-sqlite/go1/sqlite3/:534 +0x43 fp=0xc20808fcf8 sp=0xc20808fcd8
code.google.com/p/go-sqlite/go1/sqlite3.newStmt(0xc2080482d0, 0x599e90, 0x26, 0xc20804b8c8, 0x0, 0x0)
    /home/eumen/Programming/gocode/src/code.google.com/p/go-sqlite/go1/sqlite3/sqlite3.go:583 +0xe4 fp=0xc20808fdb0 sp=0xc20808fcf8
code.google.com/p/go-sqlite/go1/sqlite3.(*Conn).Query(0xc2080482d0, 0x599e90, 0x26, 0xc20808fee0, 0x1, 0x1, 0xc20807a000, 0x0, 0x0)
    /home/eumen/Programming/gocode/src/code.google.com/p/go-sqlite/go1/sqlite3/sqlite3.go:276 +0xc3 fp=0xc20808fe00 sp=0xc20808fdb0
main.query(0xc208022020, 0x16)
    /home/eumen/Programming/Go/practice/sqlite-sync.go:26 +0x261 fp=0xc20808ff70 sp=0xc20808fe00
main.main()
    /home/eumen/Programming/Go/practice/sqlite-sync.go:11 +0x66 fp=0xc20808ff98 sp=0xc20808ff70
runtime.main()
    /usr/lib/go/src/runtime/proc.go:63 +0xf3 fp=0xc20808ffe0 sp=0xc20808ff98
runtime.goexit()
    /usr/lib/go/src/runtime/asm_amd64.s:2232 +0x1 fp=0xc20808ffe8 sp=0xc20808ffe0

goroutine 2 [force gc (idle)]:
runtime.gopark(0x496870, 0x831d30, 0x583f70, 0xf)
    /usr/lib/go/src/runtime/proc.go:130 +0x105 fp=0xc20801e798 sp=0xc20801e768
runtime.goparkunlock(0x831d30, 0x583f70, 0xf)
    /usr/lib/go/src/runtime/proc.go:136 +0x48 fp=0xc20801e7c0 sp=0xc20801e798
runtime.forcegchelper()
    /usr/lib/go/src/runtime/proc.go:99 +0xce fp=0xc20801e7e0 sp=0xc20801e7c0
runtime.goexit()
    /usr/lib/go/src/runtime/asm_amd64.s:2232 +0x1 fp=0xc20801e7e8 sp=0xc20801e7e0
created by runtime.init&#183;4
    /usr/lib/go/src/runtime/proc.go:87 +0x25

goroutine 3 [GC sweep wait]:
runtime.gopark(0x496870, 0x838ff8, 0x580b50, 0xd)
    /usr/lib/go/src/runtime/proc.go:130 +0x105 fp=0xc208021f98 sp=0xc208021f68
runtime.goparkunlock(0x838ff8, 0x580b50, 0xd)
    /usr/lib/go/src/runtime/proc.go:136 +0x48 fp=0xc208021fc0 sp=0xc208021f98
runtime.bgsweep()
    /usr/lib/go/src/runtime/mgc0.go:98 +0xbc fp=0xc208021fe0 sp=0xc208021fc0
runtime.goexit()
    /usr/lib/go/src/runtime/asm_amd64.s:2232 +0x1 fp=0xc208021fe8 sp=0xc208021fe0
created by gc
    /usr/lib/go/src/runtime/mgc0.c:1386

goroutine 4 [finalizer wait]:
runtime.gopark(0x496870, 0x838ff0, 0x583ad0, 0xe)
    /usr/lib/go/src/runtime/proc.go:130 +0x105 fp=0xc20801d730 sp=0xc20801d700
runtime.goparkunlock(0x838ff0, 0x583ad0, 0xe)
    /usr/lib/go/src/runtime/proc.go:136 +0x48 fp=0xc20801d758 sp=0xc20801d730
runtime.runfinq()
    /usr/lib/go/src/runtime/malloc.go:727 +0xba fp=0xc20801d7e0 sp=0xc20801d758
runtime.goexit()
    /usr/lib/go/src/runtime/asm_amd64.s:2232 +0x1 fp=0xc20801d7e8 sp=0xc20801d7e0
created by runtime.createfing
    /usr/lib/go/src/runtime/malloc.go:707 +0x5e

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
    /usr/lib/go/src/runtime/asm_amd64.s:2232 +0x1 fp=0xc208032fe8 sp=0xc208032fe0
signal: aborted (core dumped)

This error only occurs when I increase GOMAXPROCS, and also occurs if I use go query(&amp;wg, i). Oddly enough, the errors also do not occur if the loop isn't used, or if i is set to a small enough number. Would someone be able to shed some light about what is going on here?

答案1

得分: 2

您正在使用的库不支持并发操作,根据https://code.google.com/p/go-sqlite/source/browse/go1/sqlite3/doc.go#29的说明:

并发性
单个连接实例及其所有派生对象(预编译语句、备份操作等)在没有外部同步的情况下不能从多个goroutine并发使用。唯一的例外是Conn.Interrupt(),可以从另一个goroutine中调用它来中断长时间运行的操作。即使访问同一个数据库文件,同时使用单独的连接实例也是安全的。

由于sqlite3是基于文件的数据库,驱动程序需要管理多个线程的并发访问。

https://github.com/mattn/go-sqlite3库应该支持线程安全操作(https://github.com/mattn/go-sqlite3/blob/master/sqlite3.go#L280),前提是您的sqlite3编译时支持线程安全。

英文:

The library you are using doesn't support concurrency as per https://code.google.com/p/go-sqlite/source/browse/go1/sqlite3/doc.go#29 -

> Concurrency
>
> A single connection instance and all of its derived objects (prepared
statements, backup operations, etc.) may NOT be used concurrently from multiple
goroutines without external synchronization. The only exception is
Conn.Interrupt(), which may be called from another goroutine to abort a
long-running operation. It is safe to use separate connection instances
concurrently, even if they are accessing the same database file.

As sqlite3 is a file-based database, the driver needs to manage concurrent access from multiple threads.

The https://github.com/mattn/go-sqlite3 library should support thread-safe operation provided your sqlite3 was compiled to do so.

huangapple
  • 本文由 发表于 2015年6月29日 11:22:34
  • 转载请务必保留本文链接:https://go.coder-hub.com/31107290.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定