How to organize project with multiple Go modules and use Docker compose?

I'm new to Go and am confused about how to structure a project with multiple modules of my own and use them with docker compose. The project is currently a single git repository containing two modules: an API and a processor for long-running work, each running in their own container. RabbitMQ is used as a work queue between the two and there's a MySQL database as well.

A simplified view of the project structure:

├── api
│   ├── Dockerfile
│   └── stuff.go
├── docker-compose.yml
├── go.work
└── processor
    ├── Dockerfile
    └── stuff.go

API Dockerfile (the processor one is identical except for "api" is "processor"):

FROM golang:1.20

WORKDIR /usr/src/app

COPY go.mod go.sum ./
RUN go mod download && go mod verify

COPY . .
RUN go build -v -o /usr/local/bin/ ./...

CMD [ "api" ]

Simplified docker-compose.yml:

version: '3.8'

services:
  api:
    build:
      context: ./api
	  ...

  processor:
    build:
      context: ./processor
	  ...

  rabbitmq:
    image: rabbitmq:3.11-management
	...

  mysql:
    image: mysql:8.0
	...

I run this with docker compose up --build and it works fine. The problem is there's some duplicate code in both modules that belongs in a shared module, but I don't know how to do that while still being able to build and work locally.

From researching this I get the impression that my modules should be in their own repositories. If I do that, it seems like I'd need to create a "project" repository just for the docker-compose.yml, then another for the shared module. I'd then update the docker-compose.yml contexts to point to wherever I cloned the API and processor projects when working locally.

The problem is the shared module wouldn't have a Dockerfile or be a service in the docker-compose.yml. I assume Docker would go get the contents of the shared module when building, but I still want to be able to work on that code locally without having to commit and push it to origin first.

I would start off ignoring Docker here.

Arrange your filesystem tree however you would for a normal Go application, however you'd plan to package and distribute it. You should be able to do your development with normal go test and go build commands. It sounds like you have multiple top-level commands surrounding a shared code base; putting them in a single repository makes sense, but remember that each top-level command needs its own directory.

If you have a setup like this and you want to run it in Docker, the important detail is that you need to pass the entire source tree as the build-context directory. A Dockerfile for each application is fine, and there's an argument for putting it in the same directory as its top-level application. But when you declare it in Compose you must tell it the project root directory is the build context.

I'd put the docker-compose.yml file in the project root, next to the go.mod file. It might look like:

version: '3.8'
services:
  api:
    build:
      context: .
      dockerfile: ./cmd/api/Dockerfile
    ports: ['8000:8000']
  processor: { ... }
  rabbitmq: { ... }
  mysql: { ... }

Your Dockerfiles look essentially like what you've already shown. Since the build context is the project root, all COPY commands start relative to the project root; on the other hand, you pretty much need to copy in the entire source tree no matter which application is building. For Go I would almost always use a multi-stage build to avoid including the toolchain in the final image.

# cmd/api/Dockerfile; but built with a context of the project root
FROM golang:1.20 AS builder

WORKDIR /usr/src/app

COPY go.mod go.sum ./
RUN go mod download && go mod verify

COPY . .
RUN go build -o api ./cmd/api

FROM ubuntu:22.04  # "alpine" or "not alpine" must match build stage
COPY --from=builder /usr/src/app/api /usr/local/bin/
CMD ["api"]