英文:
getting issue in querying from clickhouse using golang script for max_query_size exceeded
问题
在从ClickHouse查询时遇到了错误。
一次传递了超过20万条信息，但是从ClickHouse获取错误。
*语法错误：在位置262140处失败（''suleuha.net''）：'suleuha.net'，'diagonsys.com'，'scala.com'，'omegahns.com'，'jagrannewmedia.com'，'7ty7.com'，'bitseducampus.ac.in'，'creationspo.com'，'mafoirandstod.com'，'c.查询大小超过限制：''suleuha.net''
func errorLog(err error){
log.Errorf(err)
}
func connect(host, port) (*sql.DB, error){
connect, err := sql.Open("clickhouse", "tcp://"+host+":"+port+ "?debug=false")
connect.SetMaxOpenConns(5)
connect.SetMaxIdleConns(5)
return connect, err
}
func query() *sql.Rows{
query := fmt.Sprint("SELECT entry FROM db.suppressdomains WHERE entry IN ('" + domains + "') and scat != 'ignore'")
result, err := clickdb.Query(query)
errorLog(err)
return result
}
英文:
Getting issue in querying from clickhouse getting errror.
In one go I am passing more than 200 thousand information but getting error from clickhouse
*Syntax error: failed at position 262140 (''suleuha.net''): 'suleuha.net', 'diagonsys.com', 'scala.com', 'omegahns.com', 'jagrannewmedia.com', '7ty7.com', 'bitseducampus.ac.in', 'creationspo.com', 'mafoirandstod.com', 'c. Max query size exceeded: ''suleuha.net''
func errorLog(err error){
log.Errorf(err)
}
func connect(host, port) (*sql.DB, error){
connect, err := sql.Open("clickhouse", "tcp://"+host+":"+port+ "?debug=false")
connect.SetMaxOpenConns(5)
connect.SetMaxIdleConns(5)
return connect, err
}
func query() *sql.Rows{
query := fmt.Sprint("SELECT entry FROM db.suppressdomains WHERE entry IN ('" + domains + "') and scat != 'ignore'")
result, err := clickdb.Query(query)
errorLog(err)
return result
}
答案1
得分: 1
我在Java Clickhouse驱动程序上遇到了同样的问题。我不知道Go驱动程序是如何实现的，但很可能是因为你达到了HTTP协议的限制。
这是因为Clickhouse的HTTP接口从HTTP请求的URL部分读取请求。头部有一个最大大小限制。
在我的情况下，我尝试使用数千个列进行INSERT INTO ... (..)操作。Clickhouse Java驱动程序的修复允许将整个请求放入HTTP正文而不是HTTP头部。HTTP POST正文没有限制。
在你的情况下，我建议你改变你的方法，每次发送所有这些字符串似乎是一个糟糕的设计。你可以将这些值插入Clickhouse中的一个表中。
但同时，在Go项目的GitHub上创建一个问题报告。请求必须在正文中，而不是头部。
这里有一个示例：https://clickhouse.com/docs/en/interfaces/http/#:~:text=%24%20echo%20%27SELECT%201%27%20%7C%20curl%20%27http%3A//localhost%3A8123/%27%20%2D%2Ddata%2Dbinary%20%40%2D%0A1%0A%0A%24%20echo%20%27SELECT%201%27%20%7C%20curl%20%27http%3A//localhost%3A8123/%3Fquery%3D%27%20%2D%2Ddata%2Dbinary%20%40%2D%0A1%0A%0A%24%20echo%20%271%27%20%7C%20curl%20%27http%3A//localhost%3A8123/%3Fquery%3DSELECT%27%20%2D%2Ddata%2Dbinary%20%40%2D%0A1
英文:
I faced the same problem on the Java Clickhouse driver.
I don't know how is made the Go driver, but it is most likely that you reached the limit of the HTTP protocol.
This is because Clickhouse HTTP interface is reading the request from the URL part of the HTTP request. The header has a maximum size.
In my case, I was trying to INSERT INTO ... (..) with thousands of columns. A fix in the Clickhouse Java driver permitted to put the whole request in the HTTP body instead of the HTTP header.
The HTTP POST body has no limit.
In you case, I suggest to change your approach, sending each time all those strings seems a bad design.
You can insert those values in a table in Clickhouse.
But also create a ticket on the github of the Go project. The request have to be in the body, not in the header.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论