使用AWS Redis集群与Celery的正确方式

huangapple go评论77阅读模式
英文:

A proper way to use AWS Redis Cluster with Celery

问题

以下是您提供的内容的翻译:

我在AWS上有一个Redis集群,并且正在尝试将该Redis用作Celery的代理。
我遇到的错误是:

ValueError: 无法导入'<module_name>':无法将端口转换为整数值,因为'<redis_host>:6379'

以下是用法:

REDIS_ENDPOINT = config('REDIS_ENDPOINT')  # 应该是'hostname:port'的形式
REDIS_HOST, REDIS_PORT = REDIS_ENDPOINT.split(':')
redis_url = f'redis://{REDIS_HOST}:{REDIS_PORT}';

app.conf.update(
    broker_url=redis_url,
    # worker_concurrency=1,
    worker_prefetch_multiplier=worker_prefetch_multiplier,
)

我真的不明白为什么它不起作用。
顺便说一下,不需要密码。我100%确定redis_host的格式是有效的,因为另一个Django应用程序可以使用相同的技术轻松连接。

Django后端中的工作用法:

REDIS_HOST, REDIS_PORT = REDIS_ENDPOINT.split(':')

CACHES = {
    'default': {
        'BACKEND': 'django_redis.cache.RedisCache',
        'LOCATION': f'redis://{REDIS_HOST}:{REDIS_PORT}/0',
        'OPTIONS': {
            'CLIENT_CLASS': 'django_redis.client.DefaultClient',
        },
        'KEY_PREFIX': 'my_cache',
        'TIMEOUT': CACHE_TTL,  # 使用CACHE_TTL设置进行缓存超时
    }
}

请不要忘记联系我!感谢任何帮助!

我还尝试在Python中使用redis-py-cluster库:

REDIS_ENDPOINT = config('REDIS_ENDPOINT')  # 应该是'hostname:port'的形式
REDIS_HOST, REDIS_PORT = REDIS_ENDPOINT.split(':')
redis_broker = Redis(host=REDIS_HOST, port=REDIS_PORT)
REDIS_NODES = [{"host": REDIS_HOST, "port": REDIS_PORT}]

app.conf.update(
    broker=redis_broker,
    # result_backend=result_backend,
    broker_url='redis://',
    broker_transport_options={
        'master_name': 'mymaster',
        'startup_nodes': REDIS_NODES
    },    # result_backend=result_backend,
    # worker_concurrency=1,
    worker_prefetch_multiplier=worker_prefetch_multiplier,
)

不起作用。

我还尝试只使用redis库。在尝试第一种方法时也出现了相同的错误。

REDIS_ENDPOINT = config('REDIS_ENDPOINT')  # 应该是'hostname:port'的形式
REDIS_HOST, REDIS_PORT = REDIS_ENDPOINT.split(':')
result_backend = f'redis://{REDIS_HOST}:{REDIS_PORT}/0'
broker_url = f'redis://{REDIS_HOST}:{REDIS_PORT}/0'
redis_broker = Redis(host=REDIS_HOST, port=REDIS_PORT)

app.conf.update(
    broker=redis_broker,
    result_backend=result_backend,
    # result_backend=result_backend,
    # worker_concurrency=1,
    worker_prefetch_multiplier=worker_prefetch_multiplier,
)

希望这有助于解决您的问题!

英文:

I have a Redis cluster in AWS, and I am trying to use that Redis as a broker with Celery.
The error that I have is:

ValueError: Couldn&#39;t import &#39;&lt;module_name&gt;&#39;: Port could not be cast to integer value as &#39;&lt;redis_host&gt;:6379&#39;

Here is the usage:

REDIS_ENDPOINT = config(&#39;REDIS_ENDPOINT&#39;)  # should be of the form &#39;hostname:port&#39;
REDIS_HOST, REDIS_PORT = REDIS_ENDPOINT.split(&#39;:&#39;)
redis_url = f&#39;redis://{REDIS_HOST}:{REDIS_PORT}&#39;

app.conf.update(
    broker_url=redis_url,
    # worker_concurrency=1,
    worker_prefetch_multiplier=worker_prefetch_multiplier,
)

I really don't understand why it is not working.
No need password btw. %100 sure the redis_host is in valid format because another Django app can connect easily with the same technic.

The working usage in Django backend:

REDIS_HOST, REDIS_PORT = REDIS_ENDPOINT.split(&#39;:&#39;)

CACHES = {
    &#39;default&#39;: {
        &#39;BACKEND&#39;: &#39;django_redis.cache.RedisCache&#39;,
        &#39;LOCATION&#39;: f&#39;redis://{REDIS_HOST}:{REDIS_PORT}/0&#39;,
        &#39;OPTIONS&#39;: {
            &#39;CLIENT_CLASS&#39;: &#39;django_redis.client.DefaultClient&#39;,
        },
        &#39;KEY_PREFIX&#39;: &#39;my_cache&#39;,
        &#39;TIMEOUT&#39;: CACHE_TTL,  # Use the CACHE_TTL setting for cache timeout
    }
}

Please do not forget to ping me ! Thanks for any help!

I also tried using redis-py-cluster lib in python:

REDIS_ENDPOINT = config(&#39;REDIS_ENDPOINT&#39;)  # should be of the form &#39;hostname:port&#39;
REDIS_HOST, REDIS_PORT = REDIS_ENDPOINT.split(&#39;:&#39;)
redis_broker = Redis(host=REDIS_HOST, port=REDIS_PORT)
REDIS_NODES = [{&quot;host&quot;: REDIS_HOST, &quot;port&quot;: REDIS_PORT}]

app.conf.update(
    broker=redis_broker,
    # result_backend=result_backend,
    broker_url=&#39;redis://&#39;,
    broker_transport_options={
        &#39;master_name&#39;: &#39;mymaster&#39;,
        &#39;startup_nodes&#39;: REDIS_NODES
    },    # result_backend=result_backend,
    # worker_concurrency=1,
    worker_prefetch_multiplier=worker_prefetch_multiplier,
)

Not working.

I also used just redis lib. And also had the same error that I have when trying the first way.


REDIS_ENDPOINT = config(&#39;REDIS_ENDPOINT&#39;)  # should be of the form &#39;hostname:port&#39;
REDIS_HOST, REDIS_PORT = REDIS_ENDPOINT.split(&#39;:&#39;)
result_backend = f&#39;redis://{REDIS_HOST}:{REDIS_PORT}/0&#39;
broker_url = f&#39;redis://{REDIS_HOST}:{REDIS_PORT}/0&#39;
redis_broker = Redis(host=REDIS_HOST, port=REDIS_PORT)

app.conf.update(
    broker=redis_broker,
    result_backend=result_backend,
    # result_backend=result_backend,
    # worker_concurrency=1,
    worker_prefetch_multiplier=worker_prefetch_multiplier,
)

答案1

得分: 1

I use Celery in dev and run a Redis Cluster on k8s (not AWS) so I'm interested in a solution as well for prod. The Celery team has been working on finding a solution since 2015.

根据最近的Celery Github讨论,Celery团队目前没有资金来添加Redis Cluster支持。一名团队成员似乎取得了一些进展,但将剩余工作延迟到了v6+?

在GH上有几个关于其他项目在Redis Cluster支持上停滞不前的讨论。

这是一个声称可以启用Celery后端支持连接到Redis Cluster的存储库。没有提到Celery代理支持。

安装celery-redis-cluster-backend包后,它在celeryconfig.py中使用以下设置。

CELERY_RESULT_BACKEND = "celery_redis_cluster_backend.redis_cluster.RedisClusterBackend"
CELERY_REDIS_CLUSTER_SETTINGS = { 'startup_nodes': [
    {"host": "localhost", "port": "6379"},
    {"host": "localhost", "port": "6380"},
    {"host": "localhost", "port": "6381"}
]}
英文:

I use Celery in dev and run a Redis Cluster on k8s (not AWS) so I'm interested in a solution as well for prod. The Celery team has been working on finding a solution since 2015.

According to a recent Celery Github discussion the Celery team doesn't have the funds currently to work on adding Redis Cluster support. A team member apparently made some progress but then delayed the remaining work to v6+?

There are several discussions on GH about other projects stalled on Redis Cluster support.

Here is a repo that claims to enable Celery backend support for connecting to a Redis Cluster. No mention of Celery broker support though.

After installing the celery-redis-cluster-backend package it uses the following for celeryconfig.py.

CELERY_RESULT_BACKEND = &quot;celery_redis_cluster_backend.redis_cluster.RedisClusterBackend&quot;
CELERY_REDIS_CLUSTER_SETTINGS = { &#39;startup_nodes&#39;: [
    {&quot;host&quot;: &quot;localhost&quot;, &quot;port&quot;: &quot;6379&quot;},
    {&quot;host&quot;: &quot;localhost&quot;, &quot;port&quot;: &quot;6380&quot;},
    {&quot;host&quot;: &quot;localhost&quot;, &quot;port&quot;: &quot;6381&quot;}
]}

huangapple
  • 本文由 发表于 2023年4月13日 22:37:57
  • 转载请务必保留本文链接:https://go.coder-hub.com/76006734.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定