如何使rows.Scan更高效?

huangapple go评论85阅读模式
英文:

How to make rows.Scan more efficient?

问题

我有一个用Go编写的Web应用程序,该应用程序对Postgres数据库进行查询。当我获取到记录后,我使用rows.Next迭代记录,并使用rows.Scan将每一行扫描到一个结构体中。

如何使整个过程更快?

我认为这个程序不太高效,因为随着每个新记录的加入,扫描所有记录的时间也会增加。我考虑使用goroutine,但我担心可能会有两个goroutine扫描相同的数据。我可以通过使用互斥锁来防止这种情况发生吗?但是,如果我们使用互斥锁阻止其他goroutine访问数据,使用并发的意义何在呢?

这是我计划改进的代码:

func GetUsers() ([]publicUser, error) {
	query := `select user_id, first_name, last_name, registered_at from users;`
	rows, err := db.Query(query)
	if err != nil {
		return nil, err
	}

	var us []publicUser

	for rows.Next() {
		var u publicUser
		if err = rows.Scan(&u.UserId, &u.FirstName, &u.LastName, &u.RegisteredAt); err != nil {
			log.Println(err, "GetUsers")
			return nil, err
		}
		us = append(us, u)
	}
	if err := rows.Err(); err != nil {
		log.Println(err, "GetUsers2")
		return nil, err
	}
	return us, nil
}

我应该像这样启动一个新的goroutine吗?:

func GetUsers() ([]publicUser, error) {
	query := `select user_id, first_name, last_name, registered_at from users;`
	rows, err := db.Query(query)
	if err != nil {
		return nil, err
	}

	var us []publicUser

	for rows.Next() {
		go func() {
             var u publicUser
		     if err = rows.Scan(&u.UserId, &u.FirstName, &u.LastName, &u.RegisteredAt); err != nil 
             {
			     log.Println(err, "GetUsers")
			     return nil, err
		     }
		     us = append(us, u)
       }()
	}
	if err := rows.Err(); err != nil {
		log.Println(err, "GetUsers2")
		return nil, err
	}
	return us, nil
}

注意:在你的代码中,我将&替换为&,因为它们是相同的。

英文:

I have a web application written in Go and this application makes queries to a Postgres database. When I get back my records, I am iterating over the records with rows.Next, and scanning every row to a struct with rows.Scan.

How can I make this whole process faster?

I think this program is not very efficient because with every new record to the database, the time to scan all the records will grow as well. I thought about using goroutines, but I am worried that maybe two goroutines will scan the same data. Can I prevent this by using Mutexes? But what is the point of using concurrency, if we are preventing other goroutines from accessing the data by using mutex locks?

Here is the code I am planning to improve:

func GetUsers() ([]publicUser, error) {
	query := `select user_id, first_name, last_name, registered_at from users;`
	rows, err := db.Query(query)
	if err != nil {
		return nil, err
	}

	var us []publicUser

	for rows.Next() {
		var u publicUser
		if err = rows.Scan(&u.UserId, &u.FirstName, &u.LastName, &u.RegisteredAt); err != nil {
			log.Println(err, "GetUsers")
			return nil, err
		}
		us = append(us, u)
	}
	if err := rows.Err(); err != nil {
		log.Println(err, "GetUsers2")
		return nil, err
	}
	return us, nil
}

Should I fire a new goroutine like this ?:

func GetUsers() ([]publicUser, error) {
	query := `select user_id, first_name, last_name, registered_at from users;`
	rows, err := db.Query(query)
	if err != nil {
		return nil, err
	}

	var us []publicUser

	for rows.Next() {
		go func() {
             var u publicUser
		     if err = rows.Scan(&u.UserId, &u.FirstName, &u.LastName, &u.RegisteredAt); err != nil 
             {
			     log.Println(err, "GetUsers")
			     return nil, err
		     }
		     us = append(us, u)
       }()
	}
	if err := rows.Err(); err != nil {
		log.Println(err, "GetUsers2")
		return nil, err
	}
	return us, nil
}

答案1

得分: 1

人们通常通过使用分页来解决这个问题。基本思想是客户端可以一次请求n条记录,每个后续请求返回n条记录,偏移量为n * i + 1(其中i是当前循环迭代的次数)。

例如,如果你通过前端GUI显示用户列表,你可能希望每次显示50个结果的表格,而不是显示整个数据库中的所有用户,因为那样会减慢前端的速度。当用户点击查看下一页时,你会向服务器发出新的请求,请求50个结果,偏移量为50(因为我们现在请求的是第2页)。这种方法解决了这种情况下真正的瓶颈,即数据库而不是服务器代码。请参阅此资源了解更多信息。

> 即使结果通常很小,可列出的集合也应支持分页。

还可以参考此Stack Overflow答案,了解如何对SQL查询结果进行分页的示例。

英文:

People generally solve this problem by using pagination. The basic idea is that a client can request n number of records at a time, and every subsequent request returns n records offset by n * i + 1 (where i is the loop iteration that you're currently on).

If for example you're displaying this user list via a frontend GUI, you would probably want to have a table that shows 50 results at a time rather than all the users in your entire database, as that would slow down your frontend. Whenever the user clicks to see the next page, you would make a new request to the server asking for 50 results, offset by 50 (since we're now requesting page 2). This approach addresses the true bottleneck in this scenario which is your database, not your server code. See this resource for more information.

> Listable collections should support pagination, even if results are typically small.

Also see this SO answer for an example of paging the results from a SQL query.

huangapple
  • 本文由 发表于 2021年6月15日 23:52:16
  • 转载请务必保留本文链接:https://go.coder-hub.com/67989588.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定