英文:
Bulk INSERT in Postgres in GO using pgx
问题
我正在尝试在Go中批量插入数据库键,以下是代码:
Key结构体
type tempKey struct {
keyVal string
lastKey int
}
测试键
data := []tempKey{
{keyVal: "abc", lastKey: 10},
{keyVal: "dns", lastKey: 11},
{keyVal: "qwe", lastKey: 12},
{keyVal: "dss", lastKey: 13},
{keyVal: "xcmk", lastKey: 14},
}
插入部分
dbUrl := "数据库URL..."
conn, err := pgx.Connect(context.Background(), dbUrl)
if err != nil {
println("错误...")
}
defer conn.Close(context.Background())
sqlStr := "INSERT INTO keys (keyval,lastval) VALUES "
dollars := ""
vals := []interface{}{}
count := 1
for _, row := range data {
dollars = fmt.Sprintf("%s($%d, $%d),", dollars, count, count+1)
vals = append(vals, row.keyVal, row.lastKey)
count += 2
}
sqlStr += dollars
sqlStr = sqlStr[0 : len(sqlStr)-1]
fmt.Printf("%s \n", sqlStr)
_, erro := conn.Exec(context.Background(), sqlStr, vals)
if erro != nil {
fmt.Fprint(os.Stderr, "错误: \n", erro)
}
运行时出现错误:预期10个参数,但只有1个参数
正确的批量插入方式是什么?
英文:
I am trying to bulk insert keys in db in go here is the code
Key Struct
type tempKey struct {
keyVal string
lastKey int
}
Test Keys
data := []tempKey{
{keyVal: "abc", lastKey: 10},
{keyVal: "dns", lastKey: 11},
{keyVal: "qwe", lastKey: 12},
{keyVal: "dss", lastKey: 13},
{keyVal: "xcmk", lastKey: 14},
}
Insertion part
dbUrl := "db url...."
conn, err := pgx.Connect(context.Background(), dbUrl)
if err != nil {
println("Errrorr...")
}
defer conn.Close(context.Background())
sqlStr := "INSERT INTO keys (keyval,lastval) VALUES "
dollars := ""
vals := []interface{}{}
count := 1
for _, row := range data {
dollars = fmt.Sprintf("%s($%d, $%d),", dollars, count, count+1)
vals = append(vals, row.keyVal, row.lastKey)
count += 2
}
sqlStr += dollars
sqlStr = sqlStr[0 : len(sqlStr)-1]
fmt.Printf("%s \n", sqlStr)
_, erro := conn.Exec(context.Background(), sqlStr, vals)
if erro != nil {
fmt.Fprint(os.Stderr, "Error : \n", erro)
}
on running it throws error: expected 10 arguments, got 1
What is the correct way of bulk inserting.
答案1
得分: 21
你正在手动编写 SQL 语句,这是可以的,但你没有利用 pgx
,它可以帮助你(见下文)。
像这样追加 SQL 字符串可能对大型输入来说效率低下:
dollars = fmt.Sprintf("%s($%d, $%d),", dollars, count, count+1)
而且最终的值末尾有一个逗号“,”,而你实际上需要一个终止字符“;”来表示语句的结束。
顺便说一下,这行字符串截断是多余的:
sqlStr = sqlStr[0 : len(sqlStr)-1] // this is a NOOP
无论如何,在构建长字符串时最好使用更高效的方法,比如 strings.Builder。
根据 pgx
文档,可以使用 pgx.Conn.CopyFrom:
func (c *Conn) CopyFrom(tableName Identifier, columnNames []string, rowSrc CopyFromSource) (int, error)
> CopyFrom 使用 PostgreSQL 的复制协议执行批量数据插入。它返回复制的行数和一个错误。
示例用法:
rows := [][]interface{}{
{"John", "Smith", int32(36)},
{"Jane", "Doe", int32(29)},
}
copyCount, err := conn.CopyFrom(
pgx.Identifier{"people"},
[]string{"first_name", "last_name", "age"},
pgx.CopyFromRows(rows),
)
英文:
You are crafting the SQL statement by hand, which is fine, but you are not leveraging pgx
which can help with this (see below).
Appending to the SQL string like so can be inefficient for large inputs
dollars = fmt.Sprintf("%s($%d, $%d),", dollars, count, count+1)
but also the final value has a trailing ,
where instead you need a termination character ;
to indicate the end of the statement.
BTW this string truncation line is redundant:
sqlStr = sqlStr[0 : len(sqlStr)-1] // this is a NOOP
Anyway, better to use something more performant like strings.Builder when crafting long strings.
From the pgx
docs, use pgx.Conn.CopyFrom:
func (c *Conn) CopyFrom(tableName Identifier, columnNames []string, rowSrc CopyFromSource) (int, error)
> CopyFrom uses the PostgreSQL copy protocol to perform bulk data
> insertion. It returns the number of rows copied and an error.
example usage of Copy:
rows := [][]interface{}{
{"John", "Smith", int32(36)},
{"Jane", "Doe", int32(29)},
}
copyCount, err := conn.CopyFrom(
pgx.Identifier{"people"},
[]string{"first_name", "last_name", "age"},
pgx.CopyFromRows(rows),
)
答案2
得分: 15
使用批处理(https://github.com/jackc/pgx/blob/master/batch_test.go):
batch := &pgx.Batch{}
batch.Queue("insert into ledger(description, amount) values($1, $2)", "q1", 1)
batch.Queue("insert into ledger(description, amount) values($1, $2)", "q2", 2)
br := conn.SendBatch(context.Background(), batch)
英文:
use batch (https://github.com/jackc/pgx/blob/master/batch_test.go):
batch := &pgx.Batch{}
batch.Queue("insert into ledger(description, amount) values($1, $2)", "q1", 1)
batch.Queue("insert into ledger(description, amount) values($1, $2)", "q2", 2)
br := conn.SendBatch(context.Background(), batch)
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论