无法在Celery和Minio容器之间建立连接?

huangapple go评论85阅读模式
英文:

Not able to connect between Celery and Minio containers?

问题

This is the Docker Compose configuration you're using for your Django app:

  postgres:
    image: "postgres:14"
    environment:
      - POSTGRES_USER=user
      - POSTGRES_PASSWORD=password
    volumes:
      - .:/opt/myapp
    ports:
      - "5432:5432"

  redis:
    image: redis:6
    ports:
      - "6379:6379"

  minio:
    image: minio/minio
    command: minio server /export
    ports:
      - "9000:9000"
    environment:
      - MINIO_ACCESS_KEY=dev_access
      - MINIO_SECRET_KEY=dev_secret

  celery-worker:
    build:
      context: .
      dockerfile: ./Dockerfile.celery
    env_file:
      - ./docker.dev.env
    environment:
      - CELERY_BROKER_URL=redis://redis:6379
      - CELERY_RESULT_BACKEND=redis://redis:6379
    depends_on:
      - redis
      - postgres
    command: celery -A myapp worker --loglevel INFO

You mentioned that you want to make the Celery container connect to http://minio:9000/ instead of localhost. To achieve this, you should modify the docker.dev.env file as follows:

MINIO_STORAGE_ENDPOINT=minio:9000
MINIO_STORAGE_ACCESS_KEY=dev_access
MINIO_STORAGE_SECRET_KEY=dev_secret
MINIO_STORAGE_USE_HTTPS=False
MINIO_STORAGE_MEDIA_BUCKET_NAME=media-dev
MINIO_STORAGE_AUTO_CREATE_MEDIA_BUCKET=True
MINIO_STORAGE_MEDIA_URL=http://minio:9000/  # Update this line
DEFAULT_FILE_STORAGE=minio_storage.storage.MinioMediaStorage

By setting MINIO_STORAGE_MEDIA_URL to http://minio:9000/, you should be able to make the Celery container connect to your Minio container correctly. However, make sure your Minio settings and access permissions are configured properly to allow access from the Celery container.

英文:

In my Django app, I'm offloading resizing of uploaded videos to a celery worker. I'm also storing all media files in Minio S3 buckets.

The resizing goes well without any errors, however Celery can't connect to the Minio container. This is the logs from Celery:

celery-worker_1  | [2023-07-19 20:51:46,076: INFO/MainProcess] Task common.tasks.resize_video[c38a90eb-793e-4190-a681-94b69eafadc8] received
celery-worker_1  | ffmpeg version 4.3.6-0+deb11u1 Copyright (c) 2000-2023 the FFmpeg developers
celery-worker_1  |   built with gcc 10 (Debian 10.2.1-6)
celery-worker_1  | ffmpeg version 4.3.6-0+deb11u1[tcp @ 0x55c7500727c0]  Copyright (c) 2000-2023 the FFmpeg developers Connection to tcp://localhost:9000 failed: Cannot assign requested address
celery-worker_1  | http://localhost:9000/event_media/test_video.mp4: Cannot assign requested address

Question is: how do I make the Celery container to connect to http://minio:9000/ rather than to localhost?


This is the docker-compose.yml file I'm using:

  postgres:
    image: "postgres:14"
    environment:
      - POSTGRES_USER=user
      - POSTGRES_PASSWORD=password
    volumes:
      - .:/opt/myapp
    ports:
      - "5432:5432"

  redis:
    image: redis:6
    ports:
      - "6379:6379"

  minio:
    image: minio/minio
    command: minio server /export
    ports:
      - "9000:9000"
    environment:
      - MINIO_ACCESS_KEY=dev_access
      - MINIO_SECRET_KEY=dev_secret

  celery-worker:
    build:
      context: .
      dockerfile: ./Dockerfile.celery
    env_file:
      - ./docker.dev.env
    environment:
      - CELERY_BROKER_URL=redis://redis:6379
      - CELERY_RESULT_BACKEND=redis://redis:6379
    depends_on:
      - redis
      - postgres
    command: celery -A myapp worker --loglevel INFO

This is contents of the docker.dev.env file:

MINIO_STORAGE_ENDPOINT=minio:9000
MINIO_STORAGE_ACCESS_KEY=dev_access
MINIO_STORAGE_SECRET_KEY=dev_secret
MINIO_STORAGE_USE_HTTPS=False
MINIO_STORAGE_MEDIA_BUCKET_NAME=media-dev
MINIO_STORAGE_AUTO_CREATE_MEDIA_BUCKET=True
MINIO_STORAGE_MEDIA_URL=http://localhost:9000/
DEFAULT_FILE_STORAGE=minio_storage.storage.MinioMediaStorage

Edit: Tried to change

MINIO_STORAGE_MEDIA_URL=http://minio:9000/

Which instead gave me this following error:

celery-worker_1  | [http @ 0x55698dd429c0] HTTP error 403 Forbidden
celery-worker_1  | http://minio:9000/event_media/test_video.mp4: Server returned 403 Forbidden (access denied)

Still scratching my head

答案1

得分: 0

经过仔细查阅https://django-minio-storage.readthedocs.io/en/latest/usage/#django-settings-configuration上的文档,特别是设置键MINIO_STORAGE_MEDIA_URL,我更新了docker.dev.env

MINIO_STORAGE_MEDIA_URL=http://localhost:9000/

MINIO_STORAGE_MEDIA_URL=http://minio:9000/media-dev

我学到的是:

  1. localhost 指向了自己的容器,因此需要使用容器名称 minio(如在 docker-compose.yml 中定义)。
  2. MINIO_STORAGE_MEDIA_URL 也应包含 minio 存储桶的名称,在这种情况下是“media-dev”,由 MINIO_STORAGE_MEDIA_BUCKET_NAME 定义。
英文:

After pouring over the documentation at https://django-minio-storage.readthedocs.io/en/latest/usage/#django-settings-configuration, and specifically the setting key MINIO_STORAGE_MEDIA_URL, I updated docker.dev.env:

From

MINIO_STORAGE_MEDIA_URL=http://localhost:9000/

To

MINIO_STORAGE_MEDIA_URL=http://minio:9000/media-dev

What I learned was:

  1. that localhost points on its own container, thus need to use the container name minio instead (as defined in docker-compose.yml).
  2. The MINIO_STORAGE_MEDIA_URL should also contain the name of the minio bucket, in this case "media-dev" as defined by MINIO_STORAGE_MEDIA_BUCKET_NAME.

huangapple
  • 本文由 发表于 2023年7月20日 18:07:00
  • 转载请务必保留本文链接:https://go.coder-hub.com/76728787.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定