Rails 7 deployment fails using capistrano with SSHKit::Runner::ExecuteError

I am trying to deploy a Rails 7 app to a Vultr VPS for the very first time using Capistrano. I followed this guide on GoRails.com. All goes well up to the final step: deploy the app using Capistrano. My setup is as follows:

• Ubuntu 22.04.2 LTS Jammy (client and server)
• Ruby 3.2.2 / Rails 7.0.4.2
• rbenv 1.2.0-59-g0704e65
• Bundler version 2.4.10
• capistrano (3.17.2)
• capistrano-bundler (2.1.0)
• capistrano-passenger (0.2.1)
• capistrano-rails (1.6.2)
• capistrano-rbenv (2.2.0)
• net-ssh (7.1.0)
• openssl (default: 3.1.0)

When running cap production deploy (with HostKey /etc/ssh/ssh_host_ed25519_key in the ssh server config file) I get this error:

verify_host_key: :secure is deprecated, use :always
Thread:0x00007f169f27a710 /home/user/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/sshkit-1.21.4/lib/sshkit/runners/parallel.rb:10 run> terminated with exception (report_on_exception is true):
/home/user/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/sshkit-1.21.4/lib/sshkit/runners/parallel.rb:15:in rescue in block (2 levels) in execute': Exception while executing as deploy@xx.xx.xx.xx: could not settle on host_key algorithm (SSHKit::Runner::ExecuteError)
Server host_key preferences: ssh-ed25519
Client host_key preferences: ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp521,ecdsa-sha2-nistp384,ecdsa-sha2-nistp256,ssh-rsa-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-rsa,rsa-sha2-256,rsa-sha2-512
from /home/user/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/sshkit-1.21.4/lib/sshkit/runners/parallel.rb:11:inblock (2 levels) in execute'
/home/user/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/net-ssh-7.1.0/lib/net/ssh/transport/algorithms.rb:410:in `negotiate': could not settle on host_key algorithm (Net::SSH::Exception)
Server host_key preferences: ssh-ed25519
Client host_key preferences: ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp521,ecdsa-sha2-nistp384,ecdsa-sha2-nistp256,ssh-rsa-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-rsa,rsa-sha2-256,rsa-sha2-512
...
(Backtrace restricted to imported tasks)
cap aborted!
SSHKit::Runner::ExecuteError: Exception while executing as deploy@xx.xx.xx.xx: could not settle on host_key algorithm
Server host_key preferences: ssh-ed25519
Client host_key preferences: ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp521,ecdsa-sha2-nistp384,ecdsa-sha2-nistp256,ssh-rsa-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-rsa,rsa-sha2-256,rsa-sha2-512

And without a HostKey variable on the server I get this error:

verify_host_key: :secure is deprecated, use :always
Thread:0x00007f0d3c68a6c8 /home/user/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/sshkit-1.21.4/lib/sshkit/runners/parallel.rb:10 run> terminated with exception (report_on_exception is true):
/home/user/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/sshkit-1.21.4/lib/sshkit/runners/parallel.rb:15:in `rescue in block (2 levels) in execute': Exception while executing as deploy@xx.xx.xx.xx: fingerprint SHA256:6EF6B30F9E557F948C402C89002C7C8A is unknown for "xx.xx.xx.xx" (SSHKit::Runner::ExecuteError)
...
(Backtrace restricted to imported tasks)
cap aborted!
SSHKit::Runner::ExecuteError: Exception while executing as deploy@xx.xx.xx.xx: fingerprint SHA256:6EF6B30F9E557F948C402C89002C7C8A is unknown for "xx.xx.xx.xx"

Caused by:
Net::SSH::HostKeyUnknown: fingerprint SHA256:6EF6B30F9E557F948C402C89002C7C8A is unknown for "xx.xx.xx.xx"

Tasks: TOP => rbenv:validate
(See full trace by running task with --trace)

I can do a password-less ssh into the server, no problem, so PubKey authentication appears to work just fine.

Does anyone know how to solve this? Many thanks for any help.

I have tried many suggestions for similar sounding errors posted here and elsewhere, including:

1. deleting known_hosts on the client and adding the server fingerprint to a new known_hosts file.
2. adding PubkeyAcceptedKeyTypes=+ed25519 to the server's ssh config file (suggested here)
3. changing all private and public keys on the client and server to use ed25519
4. switching off verify_host_key setting in ssh_options in deploy.rb

> Host Keys
> ssh-ed25519 | Require the gem ed25519
> https://github.com/net-ssh/net-ssh#host-keys

You need to install two extra dependencies for ssh-ed25579 support:

bundle add ed25519 bcrypt_pbkdf

https://github.com/net-ssh/net-ssh#install

I got similar error when deploy with capistrano to Debian 12 server.

/usr/local/bundle/gems/net-ssh-7.1.0/lib/net/ssh/verifiers/always.rb:53:in `process_cache_miss': fingerprint SHA256:JHDY3pi7IAYJQP8u5UdSSO/WfOjmqdAijpKLJJH6t0 does not match for "127.20.250.55" (Net::SSH::HostKeyMismatch)

There are 2 problems (in your case):

• your capistrano not support ed25519 (fix this by adding bundle add ed25519 bcrypt_pbkdf to your ruby)

• ssh-rsa keys are deprecated and you must use something new. ed25519 is an option.

OR, if you agree with old good rsa:
add this

HostKeyAlgorithms +ssh-rsa

PubkeyAcceptedKeyTypes +ssh-rsa

into server`s /etc/ssh/sshd_config.d/enable_rsa_keys.conf
and use

service ssh restart