无法在Celery和Minio容器之间建立连接？

huangapple
go评论121阅读模式
英文:

Not able to connect between Celery and Minio containers?

问题

This is the Docker Compose configuration you're using for your Django app:

  1.   postgres:
  2.     image: "postgres:14"
  3.     environment:
  4.       - POSTGRES_USER=user
  5.       - POSTGRES_PASSWORD=password
  6.     volumes:
  7.       - .:/opt/myapp
  8.     ports:
  9.       - "5432:5432"
  11.   redis:
  12.     image: redis:6
  13.     ports:
  14.       - "6379:6379"
  16.   minio:
  17.     image: minio/minio
  18.     command: minio server /export
  19.     ports:
  20.       - "9000:9000"
  21.     environment:
  22.       - MINIO_ACCESS_KEY=dev_access
  23.       - MINIO_SECRET_KEY=dev_secret
  25.   celery-worker:
  26.     build:
  27.       context: .
  28.       dockerfile: ./Dockerfile.celery
  29.     env_file:
  30.       - ./docker.dev.env
  31.     environment:
  32.       - CELERY_BROKER_URL=redis://redis:6379
  33.       - CELERY_RESULT_BACKEND=redis://redis:6379
  34.     depends_on:
  35.       - redis
  36.       - postgres
  37.     command: celery -A myapp worker --loglevel INFO

You mentioned that you want to make the Celery container connect to http://minio:9000/ instead of localhost. To achieve this, you should modify the docker.dev.env file as follows:

  1. MINIO_STORAGE_ENDPOINT=minio:9000
  2. MINIO_STORAGE_ACCESS_KEY=dev_access
  3. MINIO_STORAGE_SECRET_KEY=dev_secret
  4. MINIO_STORAGE_USE_HTTPS=False
  5. MINIO_STORAGE_MEDIA_BUCKET_NAME=media-dev
  6. MINIO_STORAGE_AUTO_CREATE_MEDIA_BUCKET=True
  7. MINIO_STORAGE_MEDIA_URL=http://minio:9000/  # Update this line
  8. DEFAULT_FILE_STORAGE=minio_storage.storage.MinioMediaStorage

By setting MINIO_STORAGE_MEDIA_URL to http://minio:9000/, you should be able to make the Celery container connect to your Minio container correctly. However, make sure your Minio settings and access permissions are configured properly to allow access from the Celery container.

英文:

In my Django app, I'm offloading resizing of uploaded videos to a celery worker. I'm also storing all media files in Minio S3 buckets.

The resizing goes well without any errors, however Celery can't connect to the Minio container. This is the logs from Celery:

  1. celery-worker_1  | [2023-07-19 20:51:46,076: INFO/MainProcess] Task common.tasks.resize_video[c38a90eb-793e-4190-a681-94b69eafadc8] received
  2. celery-worker_1  | ffmpeg version 4.3.6-0+deb11u1 Copyright (c) 2000-2023 the FFmpeg developers
  3. celery-worker_1  |   built with gcc 10 (Debian 10.2.1-6)
  4. celery-worker_1  | ffmpeg version 4.3.6-0+deb11u1[tcp @ 0x55c7500727c0]  Copyright (c) 2000-2023 the FFmpeg developers Connection to tcp://localhost:9000 failed: Cannot assign requested address
  5. celery-worker_1  | http://localhost:9000/event_media/test_video.mp4: Cannot assign requested address

Question is: how do I make the Celery container to connect to http://minio:9000/ rather than to localhost?

This is the docker-compose.yml file I'm using:

  1.   postgres:
  2.     image: "postgres:14"
  3.     environment:
  4.       - POSTGRES_USER=user
  5.       - POSTGRES_PASSWORD=password
  6.     volumes:
  7.       - .:/opt/myapp
  8.     ports:
  9.       - "5432:5432"
  11.   redis:
  12.     image: redis:6
  13.     ports:
  14.       - "6379:6379"
  16.   minio:
  17.     image: minio/minio
  18.     command: minio server /export
  19.     ports:
  20.       - "9000:9000"
  21.     environment:
  22.       - MINIO_ACCESS_KEY=dev_access
  23.       - MINIO_SECRET_KEY=dev_secret
  25.   celery-worker:
  26.     build:
  27.       context: .
  28.       dockerfile: ./Dockerfile.celery
  29.     env_file:
  30.       - ./docker.dev.env
  31.     environment:
  32.       - CELERY_BROKER_URL=redis://redis:6379
  33.       - CELERY_RESULT_BACKEND=redis://redis:6379
  34.     depends_on:
  35.       - redis
  36.       - postgres
  37.     command: celery -A myapp worker --loglevel INFO

This is contents of the docker.dev.env file:

  1. MINIO_STORAGE_ENDPOINT=minio:9000
  2. MINIO_STORAGE_ACCESS_KEY=dev_access
  3. MINIO_STORAGE_SECRET_KEY=dev_secret
  4. MINIO_STORAGE_USE_HTTPS=False
  5. MINIO_STORAGE_MEDIA_BUCKET_NAME=media-dev
  6. MINIO_STORAGE_AUTO_CREATE_MEDIA_BUCKET=True
  7. MINIO_STORAGE_MEDIA_URL=http://localhost:9000/
  8. DEFAULT_FILE_STORAGE=minio_storage.storage.MinioMediaStorage

Edit: Tried to change

  1. MINIO_STORAGE_MEDIA_URL=http://minio:9000/

Which instead gave me this following error:

  1. celery-worker_1  | [http @ 0x55698dd429c0] HTTP error 403 Forbidden
  2. celery-worker_1  | http://minio:9000/event_media/test_video.mp4: Server returned 403 Forbidden (access denied)

Still scratching my head

答案1

得分: 0

经过仔细查阅https://django-minio-storage.readthedocs.io/en/latest/usage/#django-settings-configuration上的文档，特别是设置键MINIO_STORAGE_MEDIA_URL，我更新了docker.dev.env：

从

  1. MINIO_STORAGE_MEDIA_URL=http://localhost:9000/

到

  1. MINIO_STORAGE_MEDIA_URL=http://minio:9000/media-dev

我学到的是：

  1. localhost 指向了自己的容器，因此需要使用容器名称 minio（如在 docker-compose.yml 中定义）。
  2. MINIO_STORAGE_MEDIA_URL 也应包含 minio 存储桶的名称，在这种情况下是“media-dev”，由 MINIO_STORAGE_MEDIA_BUCKET_NAME 定义。
英文:

After pouring over the documentation at https://django-minio-storage.readthedocs.io/en/latest/usage/#django-settings-configuration, and specifically the setting key MINIO_STORAGE_MEDIA_URL, I updated docker.dev.env:

From

  1. MINIO_STORAGE_MEDIA_URL=http://localhost:9000/

To

  1. MINIO_STORAGE_MEDIA_URL=http://minio:9000/media-dev

What I learned was:

  1. that localhost points on its own container, thus need to use the container name minio instead (as defined in docker-compose.yml).
  2. The MINIO_STORAGE_MEDIA_URL should also contain the name of the minio bucket, in this case "media-dev" as defined by MINIO_STORAGE_MEDIA_BUCKET_NAME.

huangapple
  • 本文由 发表于 2023年7月20日 18:07:00
  • 转载请务必保留本文链接：https://go.coder-hub.com/76728787.html
  • celery
  • django
  • docker
  • minio
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定