英文:
A proper way to use AWS Redis Cluster with Celery
问题
以下是您提供的内容的翻译:
我在AWS上有一个Redis集群,并且正在尝试将该Redis用作Celery的代理。
我遇到的错误是:
ValueError: 无法导入'<module_name>':无法将端口转换为整数值,因为'<redis_host>:6379'
以下是用法:
REDIS_ENDPOINT = config('REDIS_ENDPOINT') # 应该是'hostname:port'的形式
REDIS_HOST, REDIS_PORT = REDIS_ENDPOINT.split(':')
redis_url = f'redis://{REDIS_HOST}:{REDIS_PORT}';
app.conf.update(
broker_url=redis_url,
# worker_concurrency=1,
worker_prefetch_multiplier=worker_prefetch_multiplier,
)
我真的不明白为什么它不起作用。
顺便说一下,不需要密码。我100%确定redis_host的格式是有效的,因为另一个Django应用程序可以使用相同的技术轻松连接。
Django后端中的工作用法:
REDIS_HOST, REDIS_PORT = REDIS_ENDPOINT.split(':')
CACHES = {
'default': {
'BACKEND': 'django_redis.cache.RedisCache',
'LOCATION': f'redis://{REDIS_HOST}:{REDIS_PORT}/0',
'OPTIONS': {
'CLIENT_CLASS': 'django_redis.client.DefaultClient',
},
'KEY_PREFIX': 'my_cache',
'TIMEOUT': CACHE_TTL, # 使用CACHE_TTL设置进行缓存超时
}
}
请不要忘记联系我!感谢任何帮助!
我还尝试在Python中使用redis-py-cluster库:
REDIS_ENDPOINT = config('REDIS_ENDPOINT') # 应该是'hostname:port'的形式
REDIS_HOST, REDIS_PORT = REDIS_ENDPOINT.split(':')
redis_broker = Redis(host=REDIS_HOST, port=REDIS_PORT)
REDIS_NODES = [{"host": REDIS_HOST, "port": REDIS_PORT}]
app.conf.update(
broker=redis_broker,
# result_backend=result_backend,
broker_url='redis://',
broker_transport_options={
'master_name': 'mymaster',
'startup_nodes': REDIS_NODES
}, # result_backend=result_backend,
# worker_concurrency=1,
worker_prefetch_multiplier=worker_prefetch_multiplier,
)
不起作用。
我还尝试只使用redis库。在尝试第一种方法时也出现了相同的错误。
REDIS_ENDPOINT = config('REDIS_ENDPOINT') # 应该是'hostname:port'的形式
REDIS_HOST, REDIS_PORT = REDIS_ENDPOINT.split(':')
result_backend = f'redis://{REDIS_HOST}:{REDIS_PORT}/0'
broker_url = f'redis://{REDIS_HOST}:{REDIS_PORT}/0'
redis_broker = Redis(host=REDIS_HOST, port=REDIS_PORT)
app.conf.update(
broker=redis_broker,
result_backend=result_backend,
# result_backend=result_backend,
# worker_concurrency=1,
worker_prefetch_multiplier=worker_prefetch_multiplier,
)
希望这有助于解决您的问题!
英文:
I have a Redis cluster in AWS, and I am trying to use that Redis as a broker with Celery.
The error that I have is:
ValueError: Couldn't import '<module_name>': Port could not be cast to integer value as '<redis_host>:6379'
Here is the usage:
REDIS_ENDPOINT = config('REDIS_ENDPOINT') # should be of the form 'hostname:port'
REDIS_HOST, REDIS_PORT = REDIS_ENDPOINT.split(':')
redis_url = f'redis://{REDIS_HOST}:{REDIS_PORT}'
app.conf.update(
broker_url=redis_url,
# worker_concurrency=1,
worker_prefetch_multiplier=worker_prefetch_multiplier,
)
I really don't understand why it is not working.
No need password btw. %100 sure the redis_host is in valid format because another Django app can connect easily with the same technic.
The working usage in Django backend:
REDIS_HOST, REDIS_PORT = REDIS_ENDPOINT.split(':')
CACHES = {
'default': {
'BACKEND': 'django_redis.cache.RedisCache',
'LOCATION': f'redis://{REDIS_HOST}:{REDIS_PORT}/0',
'OPTIONS': {
'CLIENT_CLASS': 'django_redis.client.DefaultClient',
},
'KEY_PREFIX': 'my_cache',
'TIMEOUT': CACHE_TTL, # Use the CACHE_TTL setting for cache timeout
}
}
Please do not forget to ping me ! Thanks for any help!
I also tried using redis-py-cluster lib in python:
REDIS_ENDPOINT = config('REDIS_ENDPOINT') # should be of the form 'hostname:port'
REDIS_HOST, REDIS_PORT = REDIS_ENDPOINT.split(':')
redis_broker = Redis(host=REDIS_HOST, port=REDIS_PORT)
REDIS_NODES = [{"host": REDIS_HOST, "port": REDIS_PORT}]
app.conf.update(
broker=redis_broker,
# result_backend=result_backend,
broker_url='redis://',
broker_transport_options={
'master_name': 'mymaster',
'startup_nodes': REDIS_NODES
}, # result_backend=result_backend,
# worker_concurrency=1,
worker_prefetch_multiplier=worker_prefetch_multiplier,
)
Not working.
I also used just redis lib. And also had the same error that I have when trying the first way.
REDIS_ENDPOINT = config('REDIS_ENDPOINT') # should be of the form 'hostname:port'
REDIS_HOST, REDIS_PORT = REDIS_ENDPOINT.split(':')
result_backend = f'redis://{REDIS_HOST}:{REDIS_PORT}/0'
broker_url = f'redis://{REDIS_HOST}:{REDIS_PORT}/0'
redis_broker = Redis(host=REDIS_HOST, port=REDIS_PORT)
app.conf.update(
broker=redis_broker,
result_backend=result_backend,
# result_backend=result_backend,
# worker_concurrency=1,
worker_prefetch_multiplier=worker_prefetch_multiplier,
)
答案1
得分: 1
I use Celery in dev and run a Redis Cluster on k8s (not AWS) so I'm interested in a solution as well for prod. The Celery team has been working on finding a solution since 2015.
根据最近的Celery Github讨论,Celery团队目前没有资金来添加Redis Cluster支持。一名团队成员似乎取得了一些进展,但将剩余工作延迟到了v6+?
在GH上有几个关于其他项目在Redis Cluster支持上停滞不前的讨论。
这是一个声称可以启用Celery后端支持连接到Redis Cluster的存储库。没有提到Celery代理支持。
安装celery-redis-cluster-backend
包后,它在celeryconfig.py
中使用以下设置。
CELERY_RESULT_BACKEND = "celery_redis_cluster_backend.redis_cluster.RedisClusterBackend"
CELERY_REDIS_CLUSTER_SETTINGS = { 'startup_nodes': [
{"host": "localhost", "port": "6379"},
{"host": "localhost", "port": "6380"},
{"host": "localhost", "port": "6381"}
]}
英文:
I use Celery in dev and run a Redis Cluster on k8s (not AWS) so I'm interested in a solution as well for prod. The Celery team has been working on finding a solution since 2015.
According to a recent Celery Github discussion the Celery team doesn't have the funds currently to work on adding Redis Cluster support. A team member apparently made some progress but then delayed the remaining work to v6+?
There are several discussions on GH about other projects stalled on Redis Cluster support.
Here is a repo that claims to enable Celery backend support for connecting to a Redis Cluster. No mention of Celery broker support though.
After installing the celery-redis-cluster-backend
package it uses the following for celeryconfig.py
.
CELERY_RESULT_BACKEND = "celery_redis_cluster_backend.redis_cluster.RedisClusterBackend"
CELERY_REDIS_CLUSTER_SETTINGS = { 'startup_nodes': [
{"host": "localhost", "port": "6379"},
{"host": "localhost", "port": "6380"},
{"host": "localhost", "port": "6381"}
]}
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论