英文:
Kafka Streams state store - what kind of store to use when running in Kubernetes
问题
Kafka Streams状态存储是否与KS应用程序实例共存？例如，如果我的KS应用程序在Kubernetes pod中运行，状态存储是否位于同一pod中？在Kubernetes中，使用哪种状态存储存储更好 - RocksDB还是内存？如何在应用程序中配置状态存储的类型？
英文:
Do I understand correctly that Kafka Streams state store is co-located with the KS application instance? For example if my KS application is running in a Kubernetes pod, the state store is located in the same pod? What state store storage is better to use in Kubernetes - RocksDB or in-memory? How can the type of the state store be configured in the application?
答案1
得分: 1
这取决于您的用例 - 有时，当您的主题很小时，您可以接受内存存储。但在大多数情况下，您将默认使用持久存储。要声明一个，您可以执行以下操作：
streamsBuilder.addStateStore(
Stores.keyValueStoreBuilder(
Stores.persistentKeyValueStore(
storeName
),
keySerde,
valueSerde
)
);
如果您希望使用内存存储，请将第三行替换为 inMemoryKeyValueStore。
在k8s中运行KafkaStreams应用程序有一些注意事项。首先，仅仅一个Pod是不够的。您需要将其作为一个StatefulSet来运行。在这种情况下，您的Pod将具有一个PersistentVolumeClaim挂载在Pod下的某个路径上。最好将您的 state.dir 属性设置为指向该路径的子文件夹。这样，当您的Pod关闭时，卷将被保留，并且当Pod重新启动时，它将具有所有的存储内容。
英文:
This depends on your use case - sometimes you can accept an in-memory store when you have a small topic. However in most cases you'll default to a persistent stores. To declare one you'd do:
streamsBuilder.addStateStore(Stores.keyValueStoreBuilder(Stores.persistentKeyValueStore(storeName),keySerde,valueSerde));
If you wish for a inMemoryStore replace the third line with inMemoryKeyValueStore.
Running a KafkaStreams application in k8s has a few caveats. First of all just a pod is not enough. You'll need to run this as a stateful-set. In that case your pod will have a PersistentVolumeClaim mounted on your pod under a certain path. It's best to set your state.dir property to point at a subfolder of that path. That way when your pod shuts down the volume is retained and when the pod comes back on it will have all of its store present.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论