Next.js building in container, all .env variables which not starting with NEXT_PUBLIC_ are undefined (on the server side)

I have a Next.js app that works locally (yarn dev) using .env.local for secret vars and all working fine.

When I deploy my app to a server in a Docker container, with GitHub Actions, while creating the .env file on the fly, it appears that only variables that start with NEXT_PUBLIC_ are actually defined as normal, other variables are undefined when I console.log and check in the **server **(container) logs.

So the client side env vars are fine, server side are undefined.

Here is my GitHub Actions yml that creates the .env file:

jobs:
  build_and_push:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout the repo
        uses: actions/checkout@v2

      - name: Make envfile
        uses: SpicyPizza/create-envfile@v1.3
        with:
          envkey_NEXT_PUBLIC_JWT_EXPIRATION: ${{ vars.NEXT_PUBLIC_JWT_EXPIRATION }}
          envkey_NEXT_PUBLIC_JWT_REFRESH_TOKEN_SECRET: ${{ vars.NEXT_PUBLIC_JWT_REFRESH_TOKEN_SECRET }}
          envkey_NEXT_PUBLIC_JWT_SECRET: ${{ vars.NEXT_PUBLIC_JWT_SECRET }}
          envkey_NEXT_PUBLIC_API_BASE_URL: ${{ vars.DEV_API_BASE_URL }}
          envkey_MONGO_DB_URI: ${{ secrets.DEV_MONGO_DB_URI }}
          envkey_WEATHER_DB: ${{ vars.DEV_WEATHER_DB }}
          envkey_WILLI_WEATHER_API_KEY: ${{ secrets.WILLI_WEATHER_API_KEY }}
          file_name: .env
          fail_on_empty: false

      - name: Build container image
        run: docker build -t $(echo $REGISTRY)/$(echo $IMAGE_NAME):$(echo $GITHUB_SHA | head -c7) .

      - name: Install doctl
        uses: digitalocean/action-doctl@v2
        with:
          token: ${{ secrets.DIGITALOCEAN_ACCESS_TOKEN }}

      - name: Log in to DigitalOcean Container Registry with short-lived credentials
        run: doctl registry login --expiry-seconds 600

      - name: Remove all old images
        run: if [ ! -z "$(doctl registry repository list | grep "$(echo $IMAGE_NAME)")" ]; then doctl registry repository delete-manifest $(echo $IMAGE_NAME) $(doctl registry repository list-tags $(echo $IMAGE_NAME) | grep -o "sha.*") --force; else echo "No repository"; fi

      - name: Push image to DigitalOcean Container Registry
        run: docker push --max-concurrent-uploads 1 $(echo $REGISTRY)/$(echo $IMAGE_NAME):$(echo $GITHUB_SHA | head -c7)

      - name: Run registry garbage collection
        run: doctl registry garbage-collection start --include-untagged-manifests --force

Here is my Dockerfile:

FROM node:16-alpine as dependencies
WORKDIR /my-project
COPY package.json yarn.lock ./
    
RUN yarn install --frozen-lockfile

FROM node:16-alpine as builder
WORKDIR /my-project
COPY . .
COPY --from=dependencies /my-project/node_modules ./node_modules
COPY ./.env ./node_modules/.env
RUN yarn build

FROM node:16-alpine as runner
WORKDIR /my-project
ENV NODE_ENV production

# If you are using a custom next.config.js file, uncomment this line.
# COPY --from=builder /my-project/next.config.js ./
COPY --from=builder /my-project/public ./public
COPY --from=builder /my-project/.next ./.next
COPY --from=builder /my-project/node_modules ./node_modules
COPY --from=builder /my-project/package.json ./package.json
COPY --from=builder /my-project/_fonts /usr/local/share/fonts
COPY --from=builder /my-project/_assets ./_assets

#install fontconfig and clear cache
RUN apk add --update fontconfig 
RUN fc-cache -f -v

EXPOSE 3000
CMD ["yarn", "start"]

In the Docker file, I copy the .env file into the node_modules folder to make sure that this file is created correctly. And I can verify that all variables with and without NEXT_PUBLIC_ are having values as in my original .env.local file.

Here is an output of my log from the container which was triggered on a specific endpoint request:

### Response start here:                                                                                                                                                                    
Timestamp:  5/21/23, 9:39 AM                                                                                                                                                                
NEXT_PUBLIC_API_BASE_URL http://10.0.4.6:3000/                                                                                                                                              
MONGO_DB_URI undefined                                                                                                                                                                      
WEATHER_DB undefined                                                                                                                                                                        
WILLI_WEATHER_API_KEY undefined                                                                                                                                                            
NEXT_PUBLIC_JWT_EXPIRATION 5m                                                                                                                                                              
NEXT_PUBLIC_JWT_REFRESH_TOKEN_SECRET 7c4c1c50-3230-45bf-9eae-c9b2e401c767                                                                                                                  
NEXT_PUBLIC_JWT_SECRET dd5f3089-40c3-403d-af14-d0c228b05cb4                                                                                                                                
Delta time =  142392    

Here is the output of the .env file I copied when the Docker image was built (I modified sensitive data):

/my-project/node_modules # cat .env                                                                                                                                                        
MONGO_DB_URI=mongodb+srv://mame:reJHeDc@maev.v8ilhrq.mongodb.net/?retryWrites=true&w=majority                                                                
NEXT_PUBLIC_API_BASE_URL=http://10.0.4.6:3000/                                                                                                                                              
NEXT_PUBLIC_JWT_EXPIRATION=5m                                                                                                                                                              
NEXT_PUBLIC_JWT_REFRESH_TOKEN_SECRET=7c4c1c50-3230-45bf-9eae-c9b2e401c767                                                                                                                  
NEXT_PUBLIC_JWT_SECRET=dd5f3089-40c3-403d-af14-d0c228b05cb4                                                                                                                                
WEATHER_DB=Weather                                                                                                                                                                          
WILLI_WEATHER_API_KEY=YjQ3MDkwMGDkN2VjQ3MDkwMGZjjOD

(I modified sensitive data)

I see a variable like WILLI_WEATHER_API_KEY is visible and define when Docker is built.
But not when the application is run.

Meaning: the issue should be linked to how Next.js loads environment variables.
Environment variables are loaded into the Next.js application at build time, not at runtime, which means that any changes to the environment variables after the build process won't be reflected in the running application.

In your case, you are correctly creating a .env file with all your environment variables, but by the time this file is created and placed into the node_modules directory, the Next.js application has already been built, and it's too late for these variables to be incorporated.

Next.js has a specific way of handling environment variables:

• If you prefix the variable with NEXT_PUBLIC_, it will be available on both the server and client side.
• If it doesn't start with NEXT_PUBLIC_, it will only be available on the server side.

Additionally, these variables need to be available at build time, not runtime. This means that you must provide these variables before the yarn build command is run.
In your case, the .env file is generated and copied during the Docker build process, which is after the Next.js build process, hence they are not accessible.

To solve this issue, you need to ensure that your environment variables are available before the yarn build command is run.

Your Dockerfile would then be:

FROM node:16-alpine as dependencies
WORKDIR /my-project
COPY package.json yarn.lock ./
    
RUN yarn install --frozen-lockfile

FROM node:16-alpine as builder
WORKDIR /my-project
COPY . .
COPY --from=dependencies /my-project/node_modules ./node_modules
COPY ./.env ./.env  # copy the .env file to the root of the project
RUN yarn build

FROM node:16-alpine as runner
WORKDIR /my-project
ENV NODE_ENV production

# If you are using a custom next.config.js file, uncomment this line.
# COPY --from=builder /my-project/next.config.js ./
COPY --from=builder /my-project/public ./public
COPY --from=builder /my-project/.next ./.next
COPY --from=builder /my-project/node_modules ./node_modules
COPY --from=builder /my-project/package.json ./package.json
COPY --from=builder /my-project/_fonts /usr/local/share/fonts
COPY --from=builder /my-project/_assets ./_assets
COPY --from=builder /my-project/.env ./.env  # this is probably not necessary for runtime, but does not hurt

#install fontconfig and clear cache
RUN apk add --update fontconfig 
RUN fc-cache -f -v

EXPOSE 3000
CMD ["yarn", "start"]

You should make sure not to expose sensitive data in your logs or Docker image for security reasons.
Using secrets management systems or environment variable injection at runtime (for non-Next.js variables) would be a more secure approach.