英文:
Inserting Rows in MySQL in Go very slow?
问题
所以我一直在将一个旧的PHP系统重写为Go,希望能提高性能,但是我没有得到任何改进。问题似乎出现在我向MySQL插入数据的地方。
在PHP中,对CSV文件进行一些处理、进行一些哈希计算并将大约10,000行数据插入MySQL需要40秒(未经优化的代码)。
而在Go中,只进行相同的插入10,000行(空行)的操作,没有任何处理,需要110秒。
这两个测试在同一台机器上运行,我使用的是go-mysql-driver。
现在是一些Go代码:
这是非常简化的代码,即使如此,它仍然需要将近2分钟的时间,而PHP只需要不到一半的时间。
db := GetDbCon()
defer db.Close()
stmt, _ := db.Prepare("INSERT INTO ticket ( event_id, entry_id, column_headers, column_data, hash, salt ) VALUES ( ?, ?, ?, ?, ?, ? )")
for i := 0; i < 10000; i++{
//CreateTicket(columns, line, storedEvent)
StoreTicket(models.Ticket{int64(0), storedEvent.Id, int64(i),
"", "", "", "", int64(0), int64(0)}, *stmt)
}
//Extra functions
func StoreTicket(ticket models.Ticket, stmt sql.Stmt){
stmt.Exec(ticket.EventId, ticket.EntryId, ticket.ColumnHeaders, ticket.ColumnData, ticket.Hash, ticket.Salt)
}
func GetDbCon() (sql.DB) {
db, _ := sql.Open("mysql", "bla:bla@/bla")
return *db
}
那么,是我的代码有问题,还是go-mysql-driver有问题,还是PHP插入记录的速度确实很快?
==编辑==
根据要求,我使用tcpdump记录了PHP和Go的运行情况:
文件:
通过比较这两个日志,我很难得出任何结论,两者似乎在来回发送相同大小的数据包。但是使用Go(约110秒),MySQL处理请求的时间几乎是PHP(约44秒)的两倍,而且Go似乎在发送新请求之前等待的时间稍长(尽管差异很小)。
英文:
So I have been rewriting an old PHP system to Go looking for some performance gains but I'm not get any. And the problem seems to be in the Inserts i'm doing into Mysql.
So where PHP does some processing of a CSV file, does some hashing and inserts around 10k rows in MySQL it takes 40 seconds (unoptimized code).
Now Go on the other hand stripped away of any processing and just the same inserting of 10k(empty) rows takes 110 seconds.
Both tests are run on the same machine and I'm using the go-mysql-driver.
Now for some Go code:
This is extremely dumbed down code and this still takes almost 2 minutes, compared to PHP which does it in less then half.
db := GetDbCon()
defer db.Close()
stmt, _ := db.Prepare("INSERT INTO ticket ( event_id, entry_id, column_headers, column_data, hash, salt ) VALUES ( ?, ?, ?, ?, ?, ? )")
for i := 0; i < 10000; i++{
//CreateTicket(columns, line, storedEvent)
StoreTicket(models.Ticket{int64(0), storedEvent.Id, int64(i),
"", "", "", "", int64(0), int64(0)}, *stmt)
}
//Extra functions
func StoreTicket(ticket models.Ticket, stmt sql.Stmt){
stmt.Exec(ticket.EventId, ticket.EntryId, ticket.ColumnHeaders, ticket.ColumnData, ticket.Hash, ticket.Salt)
}
func GetDbCon() (sql.DB) {
db, _ := sql.Open("mysql", "bla:bla@/bla")
return *db
}
So is it my code, the go-mysql-driver or is this normal and is PHP just really fast in inserting records?
==EDIT==
As per requested, I have recorded both PHP and Go runs with tcpdump:
The files:
I have a hard time reaching any conclusions comparing the two logs, both seem to be sending the same size packets back and forth. But with Go(~110) mysql seems to almost take twice as long to process the request then with PHP(~44), also Go seems to wait slightly longer before sending a new request again(the difference is minimal though).
答案1
得分: 4
这是一个古老的问题,但还是有道理的 - 迟做总比不做好;你将得到一个好处:
将所有数据放入一个bytes.Buffer
中,以制表符分隔、以换行符结尾,并且不加引号(如果文本会引起问题,必须先进行转义)。将NULL编码为\N
。
使用http://godoc.org/github.com/go-sql-driver/mysql#RegisterReaderHandler,并在"instream"下注册一个返回该缓冲区的函数。然后,调用LOAD DATA LOCAL INFILE "Reader::instream" INTO TABLE ...
- 这是一种非常快速的将数据导入MySQL的方法(我用Go从标准输入中的文件管道上传数据时,测得的速度约为19 MB/秒,而使用MySQL命令行客户端从标准输入上传数据的速度为18 MB/秒)。
据我所知,这个驱动程序是唯一一种无需文件即可加载数据到LOAD DATA LOCAL INFILE的方法。
英文:
It's an old question but still - better late than never; you're in for a treat:
put all your data into a bytes.Buffer
as tab-separated, newline terminated and unquoted lines (if the text causes problems, it has to be escaped first). NULL has to be encoded as \N
.
Use http://godoc.org/github.com/go-sql-driver/mysql#RegisterReaderHandler and register a function returning that buffer under "instream". Next, call LOAD DATA LOCAL INFILE "Reader::instream" INTO TABLE ...
- that's a very fast way to pump data into MySQL (I measured about 19 MB/sec with Go from a file piped from stdin compared to 18 MB/sec for the MySQL command line client when uploading data from stdin).
As far as I know, that very driver is the only way to LOAD DATA LOCAL INFILE without the need of a file.
答案2
得分: 3
我注意到你没有使用事务,如果你使用的是纯粹的MySQL 5.x与InnoDB引擎,这将会对性能造成巨大的影响,因为它会在每次插入时自动提交。
func GetDbCon() (sql.DB) {
db, _ := sql.Open("mysql", "bla:bla@/bla")
return *db
}
func PrepareTx(db *db.DB,qry string) (tx *db.Tx, s *db.Stmt, e error) {
if tx,e=db.Begin(); e!=nil {
return
}
if s, e = tx.Prepare(qry);e!=nil {
tx.Close()
}
return
}
db := GetDbCon()
defer db.Close()
qry := "INSERT INTO ticket ( event_id, entry_id, column_headers, column_data, hash, salt ) VALUES ( ?, ?, ?, ?, ?, ? )"
tx,stmt,e:=PrepareTx(db,qry)
if e!=nil {
panic(e)
}
defer tx.Rollback()
for i := 0; i < 10000; i++{
ticket:=models.Ticket{int64(0), storedEvent.Id, int64(i),"", "", "", "", int64(0), int64(0)}
stmt.Exec(ticket.EventId, ticket.EntryId, ticket.ColumnHeaders, ticket.ColumnData, ticket.Hash, ticket.Salt)
// 为了避免巨大的事务
if i % 1000 == 0 {
if e:=tx.Commit();e!=nil {
panic(e)
} else {
// 每个事务只能提交一次
tx,stmt,e=PrepareTx(db,qry)
if e!=nil {
panic(e)
}
}
}
}
// 处理剩余的数据 - 还应该检查是否已经提交
if e:=tx.Commit();e!=nil {
panic(e)
}
以上是你提供的代码的翻译。
英文:
I notice you're not using a transaction, if you're a using a vanilla mysql 5.x with InnoDB this will be a huge performance drag as it will auto-commit on every insert.
func GetDbCon() (sql.DB) {
db, _ := sql.Open("mysql", "bla:bla@/bla")
return *db
}
func PrepareTx(db *db.DB,qry string) (tx *db.Tx, s *db.Stmt, e error) {
if tx,e=db.Begin(); e!=nil {
return
}
if s, e = tx.Prepare(qry);e!=nil {
tx.Close()
}
return
}
db := GetDbCon()
defer db.Close()
qry := "INSERT INTO ticket ( event_id, entry_id, column_headers, column_data, hash, salt ) VALUES ( ?, ?, ?, ?, ?, ? )"
tx,stmt,e:=PrepareTx(db,qry)
if e!=nil {
panic(e)
}
defer tx.Rollback()
for i := 0; i < 10000; i++{
ticket:=models.Ticket{int64(0), storedEvent.Id, int64(i),"", "", "", "", int64(0), int64(0)}
stmt.Exec(ticket.EventId, ticket.EntryId, ticket.ColumnHeaders, ticket.ColumnData, ticket.Hash, ticket.Salt)
// To avoid huge transactions
if i % 1000 == 0 {
if e:=tx.Commit();e!=nil {
panic(e)
} else {
// can only commit once per transaction
tx,stmt,e=PrepareTx(db,qry)
if e!=nil {
panic(e)
}
}
}
}
// Handle left overs - should also check it isn't already committed
if e:=tx.Commit();e!=nil {
panic(e)
}
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论