Loki的S3配置用于分块和索引

huangapple go评论97阅读模式
英文:

Loki s3 configuration for chunks and indexes

问题

我正在使用Helm图表“loki-distributed”在我们的Kubernetes集群上部署Loki,并尝试配置Loki以在我的S3存储桶中使用块和索引,但是我的S3存储桶上没有创建任何文件夹或文件,以下是用于该图表的“values.yaml”文件:

在“ingester”日志中,我可以看到以下内容:

在“查询前端”日志中,我有以下信息:

有什么想法吗?谢谢。

英文:

I'm deploying Loki with the helm chart loki-distributed on our Kubernetes cluster, and trying to configure Loki to use an s3 bucket for chunks and indexes, but no folder or file has been created on my s3 bucket, here is my values.yaml file used for the chart:

loki:
  ingester:
    # Disable chunk transfer which is not possible with statefulsets
    # and unnecessary for boltdb-shipper
    max_transfer_retries: 0
    chunk_idle_period: 1h
    chunk_target_size: 1536000
    max_chunk_age: 1h
  schema_config:
    configs:
    - from: 2020-05-15
      # Which store to use for the index
      store: boltdb-shipper
      # Which store to use for the chunks
      object_store: s3
      schema: v12
      index:
        prefix: loki_
        period: 24h
  storage_config:
    aws:
      bucketnames: loki-logs-k3s
      endpoint: https://s3.de.io.cloud.ovh.net/
      region: de
      access_key_id: "my access key id"
      secret_access_key: "my secret access key"
      insecure: false
      sse_encryption: false
      http_config:
        idle_conn_timeout: 90s
        response_header_timeout: 0s
        insecure_skip_verify: false

    boltdb-shipper:
      active_index_directory: var/loki/indexes
      shared_store: s3
      cache_location: var/loki/cache

    compactor:
      working_directory: /loki/compacting
      shared_store: s3

I can see in the ingester logs the following:

level=info ts=2023-08-10T08:17:39.68137629Z caller=index_set.go:107 msg="finished uploading table loki_index_19578"
level=info ts=2023-08-10T08:17:39.681383744Z caller=index_set.go:185 msg="cleaning up unwanted indexes from table loki_index_19578"
level=info ts=2023-08-10T08:17:39.681392812Z caller=index_set.go:86 msg="uploading table loki_index_19579"
level=info ts=2023-08-10T08:17:39.681451625Z caller=table.go:318 msg="handing over indexes to shipper loki_index_19578"
level=info ts=2023-08-10T08:17:39.681460522Z caller=table.go:334 msg="finished handing over table loki_index_19578"
level=info ts=2023-08-10T08:17:39.681488176Z caller=table.go:318 msg="handing over indexes to shipper loki_index_19579"
level=info ts=2023-08-10T08:17:39.681496461Z caller=table.go:334 msg="finished handing over table loki_index_19579"
level=info ts=2023-08-10T08:17:39.686223039Z caller=index_set.go:107 msg="finished uploading table loki_index_19579"
level=info ts=2023-08-10T08:17:39.686263397Z caller=index_set.go:185 msg="cleaning up unwanted indexes from table loki_index_19579"

In the Query frontend logs, I have this :

level=info ts=2023-08-09T15:20:37.966409453Z caller=main.go:108 msg="Starting Loki" version="(version=2.8.2, branch=HEAD, revision=9f809eda7)"
level=info ts=2023-08-09T15:20:37.966455362Z caller=modules.go:1184 msg="failed to initialize usage report" err="mkdir /var/loki: read-only file system"
level=info ts=2023-08-09T15:20:37.967143294Z caller=server.go:323 http=[::]:3100 grpc=[::]:9095 msg="server listening on addresses"
level=warn ts=2023-08-09T15:20:37.967632095Z caller=experimental.go:20 msg="experimental feature in use" feature="In-memory (FIFO) cache - frontend.embedded-cache"
level=info ts=2023-08-09T15:20:37.970551753Z caller=module_service.go:82 msg=initialising module=runtime-config
level=info ts=2023-08-09T15:20:37.970673898Z caller=module_service.go:82 msg=initialising module=server
level=info ts=2023-08-09T15:20:37.970668116Z caller=module_service.go:82 msg=initialising module=cache-generation-loader
level=info ts=2023-08-09T15:20:37.970812315Z caller=module_service.go:82 msg=initialising module=query-frontend-tripperware
level=info ts=2023-08-09T15:20:37.97084668Z caller=module_service.go:82 msg=initialising module=query-frontend
level=info ts=2023-08-09T15:20:37.971029372Z caller=loki.go:499 msg="Loki started"

Any idea ? Thank you

答案1

得分: 0

我无法识别我在问题中提供的配置的问题。文档在这方面并不太有帮助。但是,我成功地使用以下配置使其工作。希望这能帮到您。

P.S. 移除 ingester 块将阻止在 S3 中创建虚假文件夹(块)和索引文件夹,即使这些值在图表中设置为默认值。您需要明确添加它们。

loki:
  auth_enabled: false
  ingester:
    lifecycler:
      ring:
        kvstore:
          store: memberlist
        replication_factor: 1
    chunk_idle_period: 30m
    chunk_block_size: 262144
    chunk_encoding: snappy
    chunk_retain_period: 1m
    max_transfer_retries: 0
    wal:
      dir: /var/loki/wal

  # -- 有关如何配置模式的更多信息,请查看 https://grafana.com/docs/loki/latest/configuration/#schema_config
  schemaConfig:
    configs:
    - from: "2020-10-24"
      index:
        period: 24h
        prefix: index_
      object_store: s3
      schema: v11
      store: boltdb-shipper

  # -- 有关如何配置存储的更多信息,请查看 https://grafana.com/docs/loki/latest/configuration/#storage_config
  storageConfig:
    boltdb_shipper:
      active_index_directory: /var/loki/index
      cache_location: /var/loki/cache
      cache_ttl: 168h
      shared_store: s3
    filesystem:
      directory: /var/loki/chunks
    aws:
      s3: s3://yourAccessKeyId:yourSecretKey@region/bucketName
      s3forcepathstyle: true
      endpoint: yourS3endppoint
英文:

Well, I wasn't able to identify the issue with the configuration I provided in my question. The documentation isn't very helpful in this regard. However, I managed to make it work using the following configuration. I hope this helps.

P.S. Removing the ingester block will prevent the creation of the fake folder (chunks) and the index folder in S3, even if these values are set to their default settings in the chart. You would need to explicitly add them.

loki:
  auth_enabled: false
  ingester:
    lifecycler:
      ring:
        kvstore:
          store: memberlist
        replication_factor: 1
    chunk_idle_period: 30m
    chunk_block_size: 262144
    chunk_encoding: snappy
    chunk_retain_period: 1m
    max_transfer_retries: 0
    wal:
      dir: /var/loki/wal

  # -- Check https://grafana.com/docs/loki/latest/configuration/#schema_config for more info on how to configure schemas
  schemaConfig:
    configs:
    - from: "2020-10-24"
      index:
        period: 24h
        prefix: index_
      object_store: s3
      schema: v11
      store: boltdb-shipper

  # -- Check https://grafana.com/docs/loki/latest/configuration/#storage_config for more info on how to configure storages
  storageConfig:
    boltdb_shipper:
      active_index_directory: /var/loki/index
      cache_location: /var/loki/cache
      cache_ttl: 168h
      shared_store: s3
    filesystem:
      directory: /var/loki/chunks
    aws:
      s3: s3://yourAccessKeyId:yourSecretKey@region/bucketName
      s3forcepathstyle: true
      endpoint: yourS3endppoint

huangapple
  • 本文由 发表于 2023年8月10日 16:37:45
  • 转载请务必保留本文链接:https://go.coder-hub.com/76873980.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定