This post will talk about how SSH handles private key forwarding and how to utilize it in a common use case for a more secure and seamless experience. You may want to study up a bit before reading on..

The Variables:

  • client/local = host0 (i.e. my laptop)
  • remote host = host1 (my server)
  • secondary host = host2 (another server)
  • ssh-agent = keychain (Keychain Access, gnome-keychain, etc..)
  • tried/sent = offered for authentication

The Facts:

  • Whichever keys are loaded in your local/clientside ssh-agent/keychain, those keys will be forwarded to whichever host you define a “ForwardAgent” for. Nothing can disable this or change which keys get forwarded, not even IdentityFile or IdentitiesOnly!

  • Even if no IdentityFile is defined for the connection and you don’t even use pki authentication (yes, even with password auth), the loaded keys on the client will get forwarded if ForwardAgent is enabled.

  • When forwarding keys, those keys will be tried/sent first when connecting to a secondary host (from a remote host), even if you define an IdentityFile. However, if you define both an IdentityFile and set the IdentitiesOnly option to yes, only the key defined by IdentityFile will be used.

This creates a security problem for the following use case: you can’t utilize ForwardAgent with multiple keys meant for different purposes where you don’t want one purpose to have access to the resources provided by the other (i.e. work, personal, secret, etc..)

Example:

For instance, I’d like to maintain two keys, one for only work machines and one for my personal machines. However, if I keep both keys loaded in my local keychain and have ForwardAgent on for my work machines (i.e. work1, work2, etc..), when I log in to work1, any user with root privileges will be able to copy my forwarded agent and use it to log into both other work boxes (work2, work3, etc..) or my personal boxes :(

The Solution:

  1. Stop using ForwardAgent.
  2. Mutliple ssh-agent processes, each with a different (purpose) key loaded.
  3. Not care and walk around like you can’t be haxed:

Solution 1:

This is straightforward. Only utilize ssh-agent on boxes you feel are secure. Anyone who has root on and box you forward keys too (and of course, any box you load your keys in ssh-agent) can merely copy your agent and use it!

Solution 2:

I found this solution from Chris Down before I understood why he did it like this. Then I realized ForwardAgent’s suckyness and it all made sense :P

When launching ssh-agent, it exports a few variables that ssh-add uses to interact with it:

?[~]> ssh-agent
SSH_AUTH_SOCK=/tmp/ssh-fmF6LLhGEY9G/agent.11527; export SSH_AUTH_SOCK;
SSH_AGENT_PID=11528; export SSH_AGENT_PID;
echo Agent pid 11528;

Storing these variables in a file, named say ~/.ssh/agents/work would create a file that can be sourced to switch over to an agent meant just for your work key. So you would ssh-add ~/.ssh/id_rsa.work, type your password, and wala! Then whenever you wanted to log into work machines, you would just source that agent file to ensure ssh uses the right auth socket.

This can be made easier with a function, which Chris has already made. He also has some scripting in place to setup and register his agents and associated keys. Pretty nifty.

The fallacy here is that you have to switch the “context” (agent) when you decide you want to log into a different set of nodes.

Looking for an easier way, I pondered if I could some how set a precommand in my .ssh/config file for specific hosts to run before the connection is made so that way, I don’t have to worry about switching agents before connecting. Sadly, ssh config doesn’t allow for anything like this :(

Final Thoughts:

For now, I’ll probably just bite the bullet and use the cdown way of doing this. I am considering a way I can define agents to use for my host declarations in .ssh/config, and then just write a ssh wrapper function to parse out what agent to use, source it, and login.

Mario Loria is a builder of diverse infrastructure with modern workloads on both bare-metal and cloud platforms. He's traversed roles in system administration, network engineering, and DevOps. You can learn more about him here.