英文:
A proper way to use AWS Redis Cluster with Celery
问题
以下是您提供的内容的翻译：
我在AWS上有一个Redis集群，并且正在尝试将该Redis用作Celery的代理。
我遇到的错误是：
ValueError: 无法导入'<module_name>'：无法将端口转换为整数值，因为'<redis_host>:6379'
以下是用法：
REDIS_ENDPOINT = config('REDIS_ENDPOINT') # 应该是'hostname:port'的形式REDIS_HOST, REDIS_PORT = REDIS_ENDPOINT.split(':')redis_url = f'redis://{REDIS_HOST}:{REDIS_PORT}';app.conf.update(broker_url=redis_url,# worker_concurrency=1,worker_prefetch_multiplier=worker_prefetch_multiplier,)
我真的不明白为什么它不起作用。
顺便说一下，不需要密码。我100%确定redis_host的格式是有效的，因为另一个Django应用程序可以使用相同的技术轻松连接。
Django后端中的工作用法：
REDIS_HOST, REDIS_PORT = REDIS_ENDPOINT.split(':')CACHES = {'default': {'BACKEND': 'django_redis.cache.RedisCache','LOCATION': f'redis://{REDIS_HOST}:{REDIS_PORT}/0','OPTIONS': {'CLIENT_CLASS': 'django_redis.client.DefaultClient',},'KEY_PREFIX': 'my_cache','TIMEOUT': CACHE_TTL, # 使用CACHE_TTL设置进行缓存超时}}
请不要忘记联系我！感谢任何帮助！
我还尝试在Python中使用redis-py-cluster库：
REDIS_ENDPOINT = config('REDIS_ENDPOINT') # 应该是'hostname:port'的形式REDIS_HOST, REDIS_PORT = REDIS_ENDPOINT.split(':')redis_broker = Redis(host=REDIS_HOST, port=REDIS_PORT)REDIS_NODES = [{"host": REDIS_HOST, "port": REDIS_PORT}]app.conf.update(broker=redis_broker,# result_backend=result_backend,broker_url='redis://',broker_transport_options={'master_name': 'mymaster','startup_nodes': REDIS_NODES}, # result_backend=result_backend,# worker_concurrency=1,worker_prefetch_multiplier=worker_prefetch_multiplier,)
不起作用。
我还尝试只使用redis库。在尝试第一种方法时也出现了相同的错误。
REDIS_ENDPOINT = config('REDIS_ENDPOINT') # 应该是'hostname:port'的形式REDIS_HOST, REDIS_PORT = REDIS_ENDPOINT.split(':')result_backend = f'redis://{REDIS_HOST}:{REDIS_PORT}/0'broker_url = f'redis://{REDIS_HOST}:{REDIS_PORT}/0'redis_broker = Redis(host=REDIS_HOST, port=REDIS_PORT)app.conf.update(broker=redis_broker,result_backend=result_backend,# result_backend=result_backend,# worker_concurrency=1,worker_prefetch_multiplier=worker_prefetch_multiplier,)
希望这有助于解决您的问题！
英文:
I have a Redis cluster in AWS, and I am trying to use that Redis as a broker with Celery.
The error that I have is:
ValueError: Couldn't import '<module_name>': Port could not be cast to integer value as '<redis_host>:6379'
Here is the usage:
REDIS_ENDPOINT = config('REDIS_ENDPOINT') # should be of the form 'hostname:port'REDIS_HOST, REDIS_PORT = REDIS_ENDPOINT.split(':')redis_url = f'redis://{REDIS_HOST}:{REDIS_PORT}'app.conf.update(broker_url=redis_url,# worker_concurrency=1,worker_prefetch_multiplier=worker_prefetch_multiplier,)
I really don't understand why it is not working.
No need password btw. %100 sure the redis_host is in valid format because another Django app can connect easily with the same technic.
The working usage in Django backend:
REDIS_HOST, REDIS_PORT = REDIS_ENDPOINT.split(':')CACHES = {'default': {'BACKEND': 'django_redis.cache.RedisCache','LOCATION': f'redis://{REDIS_HOST}:{REDIS_PORT}/0','OPTIONS': {'CLIENT_CLASS': 'django_redis.client.DefaultClient',},'KEY_PREFIX': 'my_cache','TIMEOUT': CACHE_TTL, # Use the CACHE_TTL setting for cache timeout}}
Please do not forget to ping me ! Thanks for any help!
I also tried using redis-py-cluster lib in python:
REDIS_ENDPOINT = config('REDIS_ENDPOINT') # should be of the form 'hostname:port'REDIS_HOST, REDIS_PORT = REDIS_ENDPOINT.split(':')redis_broker = Redis(host=REDIS_HOST, port=REDIS_PORT)REDIS_NODES = [{"host": REDIS_HOST, "port": REDIS_PORT}]app.conf.update(broker=redis_broker,# result_backend=result_backend,broker_url='redis://',broker_transport_options={'master_name': 'mymaster','startup_nodes': REDIS_NODES}, # result_backend=result_backend,# worker_concurrency=1,worker_prefetch_multiplier=worker_prefetch_multiplier,)
Not working.
I also used just redis lib. And also had the same error that I have when trying the first way.
REDIS_ENDPOINT = config('REDIS_ENDPOINT') # should be of the form 'hostname:port'REDIS_HOST, REDIS_PORT = REDIS_ENDPOINT.split(':')result_backend = f'redis://{REDIS_HOST}:{REDIS_PORT}/0'broker_url = f'redis://{REDIS_HOST}:{REDIS_PORT}/0'redis_broker = Redis(host=REDIS_HOST, port=REDIS_PORT)app.conf.update(broker=redis_broker,result_backend=result_backend,# result_backend=result_backend,# worker_concurrency=1,worker_prefetch_multiplier=worker_prefetch_multiplier,)
答案1
得分: 1
I use Celery in dev and run a Redis Cluster on k8s (not AWS) so I'm interested in a solution as well for prod. The Celery team has been working on finding a solution since 2015.
根据最近的Celery Github讨论，Celery团队目前没有资金来添加Redis Cluster支持。一名团队成员似乎取得了一些进展，但将剩余工作延迟到了v6+？
在GH上有几个关于其他项目在Redis Cluster支持上停滞不前的讨论。
这是一个声称可以启用Celery后端支持连接到Redis Cluster的存储库。没有提到Celery代理支持。
安装celery-redis-cluster-backend包后，它在celeryconfig.py中使用以下设置。
CELERY_RESULT_BACKEND = "celery_redis_cluster_backend.redis_cluster.RedisClusterBackend"CELERY_REDIS_CLUSTER_SETTINGS = { 'startup_nodes': [{"host": "localhost", "port": "6379"},{"host": "localhost", "port": "6380"},{"host": "localhost", "port": "6381"}]}
英文:
I use Celery in dev and run a Redis Cluster on k8s (not AWS) so I'm interested in a solution as well for prod. The Celery team has been working on finding a solution since 2015.
According to a recent Celery Github discussion the Celery team doesn't have the funds currently to work on adding Redis Cluster support. A team member apparently made some progress but then delayed the remaining work to v6+?
There are several discussions on GH about other projects stalled on Redis Cluster support.
Here is a repo that claims to enable Celery backend support for connecting to a Redis Cluster. No mention of Celery broker support though.
After installing the celery-redis-cluster-backend package it uses the following for celeryconfig.py.
CELERY_RESULT_BACKEND = "celery_redis_cluster_backend.redis_cluster.RedisClusterBackend"CELERY_REDIS_CLUSTER_SETTINGS = { 'startup_nodes': [{"host": "localhost", "port": "6379"},{"host": "localhost", "port": "6380"},{"host": "localhost", "port": "6381"}]}
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论