How to connect to database once and do many queries thru a single connection using SQLX module in my Go web application?

huangapple go评论78阅读模式
英文:

How to connect to database once and do many queries thru a single connection using SQLX module in my Go web application?

问题

我的项目是一个使用Go的Web后端,使用了gin-gonic/gin模块。
我的示例仓库

PostgreSQL数据库

我正在使用一个远程托管的数据库(Render),并使用jmoiron/sqlx模块连接到PostgreSQL数据库。与本地数据库完全相同,出现了相同的错误。但与本地数据库不同的是,托管提供了个人账户中的指标。我在那里观察当前时刻的CPU使用情况、RAM使用情况和连接数。

连接

// db是包含标准*sql.DB的*sqlx.DB
db, err := sqlx.Open("postgres", fmt.Sprintf("host=%s port=%s user=%s password=%s dbname=%s sslmode=%s",
	postgresConfig.Host, postgresConfig.Port, postgresConfig.Username, postgresConfig.Password, postgresConfig.DBName, postgresConfig.SSLMode))
if err != nil {
	panic(err)
}

我调用了sqlx.Open()方法,然后将连接指针传递给所有需要使用数据库的必要函数。如预期的那样,连接成功。但是然后我期望只有一个连接,并且在指标中只有一个连接执行SQL查询。

SQL查询

// 这是数据库的使用方式
func (r *AuthPostgres) CheckSession(identifier string) (*models.Account, error) {
	var account models.Account
	if err := r.db.Get(&account, `select login, password, credits from "Account" 
         where login=$1`,
		identifier); err != nil {
		return nil, err
	}
    return &account, nil
}

实际上,这并不完全符合预期。当我的应用程序执行SQL查询时,连接数增加了。这不应该发生。

截图。连接数指标:

  1. 连接数增加
  2. 我关闭了Go应用程序

我已经尝试过的

我有一个比这个示例稍大的项目,但在这个仓库中,我已经概述了一个函数的样子以及它如何连接到数据库。我已经尝试调整以下内容:

  1. 添加或删除指针(*,&)
  2. 不同的数据库(本地主机->远程)
  3. 更改postgres配置文件中的max_connections(本地主机)
  4. 设置连接的超时时间
  5. 设置最大打开/空闲连接限制

这些对我来说都没有起作用。我希望能够获得一个单一的连接,并且每分钟执行多次SQL查询。

日志

在不同的方式下进行测试时,我从日志中收到的错误。我已经强调了最重要的部分。这些错误以不同的顺序和不同的组合重复出现。

  1. FATAL: sorry too many clients already
  2. pq: out of memory
  3. fatal error: out of memory allocated heap arena metadata
英文:

My project is web back-end that using Go http framework: gin-gonic/gin module.
My example repository

PostgreSQL database

I am using a remote hosted database (Render) and jmoiron/sqlx module to connect to PostgreSQL database. With the local database exactly the same story, the same errors. But unlike a local database, hosting provides metrics in your personal account. I observe there the use of the CPU, RAM and the number of connections at the current moment.

Connection

// db is *sqlx.DB that contain standard *sql.DB
db, err := sqlx.Open("postgres", fmt.Sprintf("host=%s port=%s user=%s password=%s dbname=%s sslmode=%s",
	postgresConfig.Host, postgresConfig.Port, postgresConfig.Username, postgresConfig.Password, postgresConfig.DBName, postgresConfig.SSLMode))
if err != nil {
	panic(err)
}

I call the sqlx.Open() method and then pass the connection pointer to all my necessary functions that use the database. As expected, the connection succeeds. But then I expected one single connection and execution of sql queries with one connection in the metric.

SQL query

// this is how database use
func (r *AuthPostgres) CheckSession(identifier string) (*models.Account, error) {
	var account models.Account
	if err := r.db.Get(&account, `select login, password, credits from "Account" 
         where login=$1`,
		identifier); err != nil {
		return nil, err
	}
    return &account, nil
}

Actually this is not quite as expected. When my application executes sql queries, the number of connections grows. This shouldn't happen.

Screenshots. Amount of connections metrics:

  1. Amount of connections grows
  2. I turned off the go application

What I already tried

I have a project that is slightly larger than this example, but in this repository I have outlined exactly what one function looks like and how it connects to the database. And I already tried to tweaking this:

  1. Add or remove pointers (*, &)
  2. Different database (localhost -> remote)
  3. Change max_connections in postgres config file (localhost)
  4. Set time-out for connection
  5. Set max open/idle connection limit

These not work for me. I would like get a single connection and do sql queries many times per minute.

Logs

Errors that I recieved from logs during testing in different ways. I have highlighted the most important. These were repeated in different order and in different combinations.

  1. FATAL: sorry too many clients already
  2. pq: out of memory
  3. fatal error: out of memory allocated heap arena metadata

答案1

得分: 1

sql.DB是该结构的一个不好的名称。它不是一个单独的连接,而是一个连接池

这是在大多数实际应用程序中所需要的。连接池可以实现并发性,并通过消除显式管理连接的需要来简化代码。

通过使用DB.SetMaxOpenConnsDB.SetMaxIdleConns方法,您仍然可以控制连接池的大小和生命周期。

英文:

sql.DB is a poor name for that struct. It is not a single connection it is a connection pool

That is what you need in the majority of real-life applications. Connection Poll enables concurrency and simplifies code by eliminating the need to manage connections explicitly.

You still have control over Connection Pool size and lifetime with DB.SetMaxOpenConns and DB.SetMaxIdleConns methods.

答案2

得分: 0

答案是:使用上下文和使用sqlx.ConnectContext()打开连接池。

-	// db是包含*sql.DB的*sqlx.DB
-	db, err := sqlx.Connect("postgres", fmt.Sprintf("host=%s port=%s user=%s password=%s dbname=%s sslmode=%s",
+	ctxTimeout, ctxCancel := context.WithTimeout(context.Background(), time.Second*3)
+	defer ctxCancel()
+
+	db, err := sqlx.ConnectContext(ctxTimeout, "postgres", fmt.Sprintf("host=%s port=%s user=%s password=%s dbname=%s sslmode=%s",
		postgresConfig.Host, postgresConfig.Port, postgresConfig.Username, postgresConfig.Password, postgresConfig.DBName, postgresConfig.SSLMode))
	if err != nil {
		panic(err)
	}

为了节省资源并控制数据库连接池中的空闲活动:

+	db.SetMaxIdleConns(5)
+	db.SetConnMaxIdleTime(10 * time.Second)
+	db.SetMaxOpenConns(95)
英文:

The answer is: Using context and open connection pool with sqlx.ConnectContext().

-	// db is *sqlx.DB that contain *sql.DB
-	db, err := sqlx.Connect("postgres", fmt.Sprintf("host=%s port=%s user=%s password=%s dbname=%s sslmode=%s",
+	ctxTimeout, ctxCancel := context.WithTimeout(context.Background(), time.Second*3)
+	defer ctxCancel()
+
+	db, err := sqlx.ConnectContext(ctxTimeout, "postgres", fmt.Sprintf("host=%s port=%s user=%s password=%s dbname=%s sslmode=%s",
		postgresConfig.Host, postgresConfig.Port, postgresConfig.Username, postgresConfig.Password, postgresConfig.DBName, postgresConfig.SSLMode))
	if err != nil {
		panic(err)
	}

To save resources and control idle activity in your database pool:

+	db.SetMaxIdleConns(5)
+	db.SetConnMaxIdleTime(10 * time.Second)
+	db.SetMaxOpenConns(95)

huangapple
  • 本文由 发表于 2022年8月15日 19:35:40
  • 转载请务必保留本文链接:https://go.coder-hub.com/73360446.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定