英文:
Loki s3 configuration for chunks and indexes
问题
我正在使用Helm图表“loki-distributed”在我们的Kubernetes集群上部署Loki,并尝试配置Loki以在我的S3存储桶中使用块和索引,但是我的S3存储桶上没有创建任何文件夹或文件,以下是用于该图表的“values.yaml”文件:
在“ingester”日志中,我可以看到以下内容:
在“查询前端”日志中,我有以下信息:
有什么想法吗?谢谢。
英文:
I'm deploying Loki with the helm chart loki-distributed
on our Kubernetes cluster, and trying to configure Loki to use an s3 bucket for chunks and indexes, but no folder or file has been created on my s3 bucket, here is my values.yaml
file used for the chart:
loki:
ingester:
# Disable chunk transfer which is not possible with statefulsets
# and unnecessary for boltdb-shipper
max_transfer_retries: 0
chunk_idle_period: 1h
chunk_target_size: 1536000
max_chunk_age: 1h
schema_config:
configs:
- from: 2020-05-15
# Which store to use for the index
store: boltdb-shipper
# Which store to use for the chunks
object_store: s3
schema: v12
index:
prefix: loki_
period: 24h
storage_config:
aws:
bucketnames: loki-logs-k3s
endpoint: https://s3.de.io.cloud.ovh.net/
region: de
access_key_id: "my access key id"
secret_access_key: "my secret access key"
insecure: false
sse_encryption: false
http_config:
idle_conn_timeout: 90s
response_header_timeout: 0s
insecure_skip_verify: false
boltdb-shipper:
active_index_directory: var/loki/indexes
shared_store: s3
cache_location: var/loki/cache
compactor:
working_directory: /loki/compacting
shared_store: s3
I can see in the ingester
logs the following:
level=info ts=2023-08-10T08:17:39.68137629Z caller=index_set.go:107 msg="finished uploading table loki_index_19578"
level=info ts=2023-08-10T08:17:39.681383744Z caller=index_set.go:185 msg="cleaning up unwanted indexes from table loki_index_19578"
level=info ts=2023-08-10T08:17:39.681392812Z caller=index_set.go:86 msg="uploading table loki_index_19579"
level=info ts=2023-08-10T08:17:39.681451625Z caller=table.go:318 msg="handing over indexes to shipper loki_index_19578"
level=info ts=2023-08-10T08:17:39.681460522Z caller=table.go:334 msg="finished handing over table loki_index_19578"
level=info ts=2023-08-10T08:17:39.681488176Z caller=table.go:318 msg="handing over indexes to shipper loki_index_19579"
level=info ts=2023-08-10T08:17:39.681496461Z caller=table.go:334 msg="finished handing over table loki_index_19579"
level=info ts=2023-08-10T08:17:39.686223039Z caller=index_set.go:107 msg="finished uploading table loki_index_19579"
level=info ts=2023-08-10T08:17:39.686263397Z caller=index_set.go:185 msg="cleaning up unwanted indexes from table loki_index_19579"
In the Query frontend
logs, I have this :
level=info ts=2023-08-09T15:20:37.966409453Z caller=main.go:108 msg="Starting Loki" version="(version=2.8.2, branch=HEAD, revision=9f809eda7)"
level=info ts=2023-08-09T15:20:37.966455362Z caller=modules.go:1184 msg="failed to initialize usage report" err="mkdir /var/loki: read-only file system"
level=info ts=2023-08-09T15:20:37.967143294Z caller=server.go:323 http=[::]:3100 grpc=[::]:9095 msg="server listening on addresses"
level=warn ts=2023-08-09T15:20:37.967632095Z caller=experimental.go:20 msg="experimental feature in use" feature="In-memory (FIFO) cache - frontend.embedded-cache"
level=info ts=2023-08-09T15:20:37.970551753Z caller=module_service.go:82 msg=initialising module=runtime-config
level=info ts=2023-08-09T15:20:37.970673898Z caller=module_service.go:82 msg=initialising module=server
level=info ts=2023-08-09T15:20:37.970668116Z caller=module_service.go:82 msg=initialising module=cache-generation-loader
level=info ts=2023-08-09T15:20:37.970812315Z caller=module_service.go:82 msg=initialising module=query-frontend-tripperware
level=info ts=2023-08-09T15:20:37.97084668Z caller=module_service.go:82 msg=initialising module=query-frontend
level=info ts=2023-08-09T15:20:37.971029372Z caller=loki.go:499 msg="Loki started"
Any idea ? Thank you
答案1
得分: 0
我无法识别我在问题中提供的配置的问题。文档在这方面并不太有帮助。但是,我成功地使用以下配置使其工作。希望这能帮到您。
P.S. 移除 ingester 块将阻止在 S3 中创建虚假文件夹(块)和索引文件夹,即使这些值在图表中设置为默认值。您需要明确添加它们。
loki:
auth_enabled: false
ingester:
lifecycler:
ring:
kvstore:
store: memberlist
replication_factor: 1
chunk_idle_period: 30m
chunk_block_size: 262144
chunk_encoding: snappy
chunk_retain_period: 1m
max_transfer_retries: 0
wal:
dir: /var/loki/wal
# -- 有关如何配置模式的更多信息,请查看 https://grafana.com/docs/loki/latest/configuration/#schema_config
schemaConfig:
configs:
- from: "2020-10-24"
index:
period: 24h
prefix: index_
object_store: s3
schema: v11
store: boltdb-shipper
# -- 有关如何配置存储的更多信息,请查看 https://grafana.com/docs/loki/latest/configuration/#storage_config
storageConfig:
boltdb_shipper:
active_index_directory: /var/loki/index
cache_location: /var/loki/cache
cache_ttl: 168h
shared_store: s3
filesystem:
directory: /var/loki/chunks
aws:
s3: s3://yourAccessKeyId:yourSecretKey@region/bucketName
s3forcepathstyle: true
endpoint: yourS3endppoint
英文:
Well, I wasn't able to identify the issue with the configuration I provided in my question. The documentation isn't very helpful in this regard. However, I managed to make it work using the following configuration. I hope this helps.
P.S. Removing the ingester block will prevent the creation of the fake folder (chunks) and the index folder in S3, even if these values are set to their default settings in the chart. You would need to explicitly add them.
loki:
auth_enabled: false
ingester:
lifecycler:
ring:
kvstore:
store: memberlist
replication_factor: 1
chunk_idle_period: 30m
chunk_block_size: 262144
chunk_encoding: snappy
chunk_retain_period: 1m
max_transfer_retries: 0
wal:
dir: /var/loki/wal
# -- Check https://grafana.com/docs/loki/latest/configuration/#schema_config for more info on how to configure schemas
schemaConfig:
configs:
- from: "2020-10-24"
index:
period: 24h
prefix: index_
object_store: s3
schema: v11
store: boltdb-shipper
# -- Check https://grafana.com/docs/loki/latest/configuration/#storage_config for more info on how to configure storages
storageConfig:
boltdb_shipper:
active_index_directory: /var/loki/index
cache_location: /var/loki/cache
cache_ttl: 168h
shared_store: s3
filesystem:
directory: /var/loki/chunks
aws:
s3: s3://yourAccessKeyId:yourSecretKey@region/bucketName
s3forcepathstyle: true
endpoint: yourS3endppoint
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论