可以多个esutil.BulkIndexer共享一个elasticsearch.Client吗?

huangapple go评论90阅读模式
英文:

Can several esutil.BulkIndexer share an elasticsearch.Client?

问题

我有一个文档数组需要使用github.com/elastic/go-elasticsearch/v8库进行批量索引。

每个文档可能被添加到少数几个不同的索引中,所以我的代码如下:

    var indexers map[string]esutil.BulkIndexer = make(map[string]esutil.BulkIndexer)
    
    for i := range documents {
      var bi esutil.BulkIndexer
	  var ok bool
	  var err error
	  if bi, ok = indexers[documents[i].index]; !ok {
		 bi, err = esutil.NewBulkIndexer(esutil.BulkIndexerConfig{
			Index:         documents[i].index,   // 索引名称
			Client:        client,               // Elasticsearch客户端*所有索引器共享同一个实例*
			NumWorkers:    BULK_NUM_WORKERS,     // 工作goroutine的数量
			FlushBytes:    int(BULK_FLUSHBYTES), // 刷新阈值(以字节为单位)
			FlushInterval: 5 * time.Second,      // 定期刷新间隔
		})
		if err != nil {
			fmt.Printf("创建索引器时出错:%s", err)
			continue
		}
		indexers[documents[i].indexName] = bi
	}

我的想法是我有一个esutil.BulkIndexer的映射,它们都共享同一个elasticsearch.Client实例。所以我的问题是,多个esutil.BulkIndexer共享同一个elasticsearch.Client实例是否安全和可行,还是它们每个都需要一个单独的elasticsearch.Client实例?

英文:

I have an array of documents to bulk indexusing the github.com/elastic/go-elasticsearch/v8 library.

Each document may be added to one of a handful of different indexes, so my code is something like:

    var indexers map[string]esutil.BulkIndexer = make(map[string]esutil.BulkIndexer)
    
    for i := range documents {
      var bi esutil.BulkIndexer
	  var ok bool
	  var err error
	  if bi, ok = indexers[documents[i].index]; !ok {
		 bi, err = esutil.NewBulkIndexer(esutil.BulkIndexerConfig{
			Index:         documents[i].index,   // The index name
			Client:        client,               // The Elasticsearch client* Same isntance for all indexers*
			NumWorkers:    BULK_NUM_WORKERS,     // The number of worker goroutines
			FlushBytes:    int(BULK_FLUSHBYTES), // The flush threshold in bytes
			FlushInterval: 5 * time.Second,      // The periodic flush interval
		})
		if err != nil {
			fmt.Printf("Error creating the indexer: %s", err)
			continue
		}
		indexers[documents[i].indexName] = bi
	}

The idea is I have a map of esutil.BulkIndexer , an they all share the same elasticsearch.Client instance. So my question is if it's ok and safe for several
esutil.BulkIndexer to share the same elasticsearch.Client, or do they each need a separate instance of a elasticsearch.Client ?

答案1

得分: 0

无法在官方文档中找到详细信息。但是根据此链接,建议在并行请求中共享客户端连接。

客户端连接是在包初始化时创建的,后续以并行方式进行的请求共享同一个客户端。

来自上述链接的示例代码如下:

package httpexample

import (
	"github.com/elastic/go-elasticsearch/v7"
)

var client *elasticsearch.Client

func init() {
	var err error

	... // 客户端配置
	client, err = elasticsearch.NewClient(cfg)
	if err != nil {
		log.Fatalf("elasticsearch.NewClient: %v", err)
	}
}

// 此方法将并行调用
func HttpExample(w http.ResponseWriter, r *http.Request) {
	... // 使用客户端
}

注意:我建议使用多个bulkIndexer。

英文:

Could not find the details on official documentation. But Looking at this link it is recommended to share the client across the workers(go routines) for parallel requests

The client connection is created on package initialization, and subsequent requests happening in parallel fashion shares the same client.

from the above link

package httpexample
import (
	"github.com/elastic/go-elasticsearch/v7"
)

var client *elasticsearch.Client

func init() {
	var err error

	... # Client configuration
	client, err = elasticsearch.NewClient(cfg)
	if err != nil {
		log.Fatalf("elasticsearch.NewClient: %v", err)
	}
}
//This method will be called in parallel
func HttpExample(w http.ResponseWriter, r *http.Request) {
	... # Client usage
}

Note: I am an expert to give a recommendation on using multiple bulkIndexer.

huangapple
  • 本文由 发表于 2021年8月20日 15:04:46
  • 转载请务必保留本文链接:https://go.coder-hub.com/68858158.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定