在Go语言中使用pgx进行Postgres的批量插入操作。

huangapple go评论206阅读模式
英文:

Bulk INSERT in Postgres in GO using pgx

问题

我正在尝试在Go中批量插入数据库键,以下是代码:

Key结构体

type tempKey struct {
    keyVal  string
    lastKey int
}

测试键

data := []tempKey{
    {keyVal: "abc", lastKey: 10},
    {keyVal: "dns", lastKey: 11},
    {keyVal: "qwe", lastKey: 12},
    {keyVal: "dss", lastKey: 13},
    {keyVal: "xcmk", lastKey: 14},
}

插入部分

dbUrl := "数据库URL..."
conn, err := pgx.Connect(context.Background(), dbUrl)
if err != nil {
    println("错误...")
}
defer conn.Close(context.Background())
sqlStr := "INSERT INTO keys (keyval,lastval) VALUES "
dollars := ""
vals := []interface{}{}
count := 1
for _, row := range data {
    dollars = fmt.Sprintf("%s($%d, $%d),", dollars, count, count+1)
    vals = append(vals, row.keyVal, row.lastKey)
    count += 2
}
sqlStr += dollars
sqlStr = sqlStr[0 : len(sqlStr)-1]
fmt.Printf("%s \n", sqlStr)

_, erro := conn.Exec(context.Background(), sqlStr, vals)
if erro != nil {
    fmt.Fprint(os.Stderr, "错误: \n", erro)
}

运行时出现错误:预期10个参数,但只有1个参数

正确的批量插入方式是什么?

英文:

I am trying to bulk insert keys in db in go here is the code
Key Struct

type tempKey struct {
keyVal  string
lastKey int

}

Test Keys

data := []tempKey{
	{keyVal: "abc", lastKey: 10},
	{keyVal: "dns", lastKey: 11},
	{keyVal: "qwe", lastKey: 12},
	{keyVal: "dss", lastKey: 13},
	{keyVal: "xcmk", lastKey: 14},
}

Insertion part

dbUrl := "db url...."
conn, err := pgx.Connect(context.Background(), dbUrl)
if err != nil {
	println("Errrorr...")
}
defer conn.Close(context.Background())
sqlStr := "INSERT INTO keys (keyval,lastval) VALUES "
dollars := ""
vals := []interface{}{}
count := 1
for _, row := range data {
	dollars = fmt.Sprintf("%s($%d, $%d),", dollars, count, count+1)
	vals = append(vals, row.keyVal, row.lastKey)
	count += 2
}
sqlStr += dollars
sqlStr = sqlStr[0 : len(sqlStr)-1]
fmt.Printf("%s \n", sqlStr)

_, erro := conn.Exec(context.Background(), sqlStr, vals)
if erro != nil {
	fmt.Fprint(os.Stderr, "Error : \n", erro)
}

on running it throws error: expected 10 arguments, got 1

What is the correct way of bulk inserting.

答案1

得分: 21

你正在手动编写 SQL 语句,这是可以的,但你没有利用 pgx,它可以帮助你(见下文)。

像这样追加 SQL 字符串可能对大型输入来说效率低下:

dollars = fmt.Sprintf("%s($%d, $%d),", dollars, count, count+1)

而且最终的值末尾有一个逗号“,”,而你实际上需要一个终止字符“;”来表示语句的结束。

顺便说一下,这行字符串截断是多余的:

sqlStr = sqlStr[0 : len(sqlStr)-1] // this is a NOOP

无论如何,在构建长字符串时最好使用更高效的方法,比如 strings.Builder


根据 pgx 文档,可以使用 pgx.Conn.CopyFrom

func (c *Conn) CopyFrom(tableName Identifier, columnNames []string, rowSrc CopyFromSource) (int, error)

> CopyFrom 使用 PostgreSQL 的复制协议执行批量数据插入。它返回复制的行数和一个错误。

示例用法

rows := [][]interface{}{
    {"John", "Smith", int32(36)},
    {"Jane", "Doe", int32(29)},
}

copyCount, err := conn.CopyFrom(
    pgx.Identifier{"people"},
    []string{"first_name", "last_name", "age"},
    pgx.CopyFromRows(rows),
)
英文:

You are crafting the SQL statement by hand, which is fine, but you are not leveraging pgx which can help with this (see below).

Appending to the SQL string like so can be inefficient for large inputs

dollars = fmt.Sprintf("%s($%d, $%d),", dollars, count, count+1)

but also the final value has a trailing , where instead you need a termination character ; to indicate the end of the statement.

BTW this string truncation line is redundant:

sqlStr = sqlStr[0 : len(sqlStr)-1] // this is a NOOP

Anyway, better to use something more performant like strings.Builder when crafting long strings.


From the pgx docs, use pgx.Conn.CopyFrom:

func (c *Conn) CopyFrom(tableName Identifier, columnNames []string, rowSrc CopyFromSource) (int, error)

> CopyFrom uses the PostgreSQL copy protocol to perform bulk data
> insertion. It returns the number of rows copied and an error.

example usage of Copy:

rows := [][]interface{}{
    {"John", "Smith", int32(36)},
    {"Jane", "Doe", int32(29)},
}

copyCount, err := conn.CopyFrom(
    pgx.Identifier{"people"},
    []string{"first_name", "last_name", "age"},
    pgx.CopyFromRows(rows),
)

答案2

得分: 15

使用批处理(https://github.com/jackc/pgx/blob/master/batch_test.go):

batch := &pgx.Batch{}
batch.Queue("insert into ledger(description, amount) values($1, $2)", "q1", 1)
batch.Queue("insert into ledger(description, amount) values($1, $2)", "q2", 2)
br := conn.SendBatch(context.Background(), batch)
英文:

use batch (https://github.com/jackc/pgx/blob/master/batch_test.go):

batch := &pgx.Batch{}
batch.Queue("insert into ledger(description, amount) values($1, $2)", "q1", 1)
batch.Queue("insert into ledger(description, amount) values($1, $2)", "q2", 2)
br := conn.SendBatch(context.Background(), batch)

huangapple
  • 本文由 发表于 2022年1月23日 22:30:29
  • 转载请务必保留本文链接:https://go.coder-hub.com/70823061.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定