SCP from GitLab pipeline fails (Permission denied)

I need to scp some files from a gitlab pipeline to a server. Here are the steps I took to accomplish this, and the error I've received.

On my local machine, I create a public and private key to use specifically for this, and nothing else. The private key is for a variable in my gitlab pipeline, and the public key goes in the server's authorized key files.:

ssh-keygen -t rsa -b 2048 -C "Pipeline"


Enter file in which to save the key (/home/user/.ssh/id_rsa): /home/user/.ssh/id_rsa_pipeline

This creates the public (id_rsa_pipeline.pub) and private (id_rsa_pipeline) key successfully.

I then add the key to the remote server:

ssh-copy-id -i id_rsa_temp user@server.com

/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "id_rsa_pipeline.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'user@server.com'"
and check to make sure that only the key(s) you wanted were added.

I then test the login, using the key ssh -i id_rsa_pipeline user@server.com, and it works as expected. No issues logging in using key, without password.

I then add a variable in my pipeline named IDENTITY.

My stage that does the work in my pipeline has a before_script like this.

  before_script:
  - 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
  - eval $(ssh-agent -s)
  - echo "$IDENTITY" | tr -d '\r' | ssh-add - > /dev/null
  - mkdir -p ~/.ssh
  - chmod 700 ~/.ssh

My scp command I'm using in the pipeline looks like this:

after_script:
    - scp -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null /path/to/my/file.txt user@development:/home/user/

When I run the pipeline, I'm getting the following:

Running after script...
$ scp -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null /path/to/my/file.txt user@development:/home/user/
Warning: Permanently added 'user,192.168.1.50' (ECDSA) to the list of known hosts.
Permission denied, please try again.
Permission denied, please try again.
user@server: Permission denied (publickey,password).
lost connection

Things I've tried:

1. Creating the keys again.
2. Making sure no whitespaces or return characters got into the key when copy/pasting into gitlab.
3. Creating the keys on different servers.
4. Using different users.
5. General googling.

On every user I've tried this with, I can ssh into the server using the key, and I can scp files without passwords using the keys. Any help is appreciated.

For testing, I would:

• create a key pair again, but without passphrase.
• create a new file type CI/CD variable. As Key, enter the name SSH_PRIVATE_KEY, and in the Value field, paste the content of your private key that you created earlier.
• follow the same process as in "SSH keys when using the Docker executor"

before_script:
  ##
  ## Install ssh-agent if not already installed, it is required by Docker.
  ## (change apt-get to yum if you use an RPM-based image)
  ##
  - 'command -v ssh-agent >/dev/null || ( apt-get update -y && apt-get install openssh-client -y )'

  ##
  ## Run ssh-agent (inside the build environment)
  ##
  - eval $(ssh-agent -s)

  ##
  ## Give the right permissions, otherwise ssh-add will refuse to add files
  ## Add the SSH key stored in SSH_PRIVATE_KEY file type CI/CD variable to the agent store
  ##
  - chmod 400 "$SSH_PRIVATE_KEY"
  - ssh-add "$SSH_PRIVATE_KEY"

  ##
  ## Create the SSH directory and give it the right permissions
  ##
  - mkdir -p ~/.ssh
  - chmod 700 ~/.ssh

  ##
  ## Optionally, if you will be using any Git commands, set the user name and
  ## and email.
  ##
  # - git config --global user.email "user@example.com"
  # - git config --global user.name "User name"

Make sure the private server’s SSH host keys are verified.

You can try adding the -o IdentityFile=/path/to/private/ssh/key over the afterScript scp command to be precise on which ssh keys has to be used.