Ok so this gets a little intense to think about but its something you might run into if you are using virtual terminals with ssh keys and agent forwarding etc..

NOTE: All code involved here is stored and updated at my github here!!

for all intents and purposes, tmux = screen for the rest of the article.

My first problem: When leaving a tmux session running on a server, logging out of that server and going home, logging back in and pulling up my tmux session (tmux attach), my key forwarding doesn’t work anymore. This is because your tmux session is using your old SSH_AUTH_SOCK values from when you logged in originally to create the tmux session. However, this ssh socket doesn’t exist anymore. A new one has been created when you logged in. This is simple to see from echo’ing $SSH_AUTH_SOCK in your tmux session and then again out of your tmux session and observing the difference.

To fix this, some of the common methods seem to be making a symlink in your home dir which gets updated with the newest socket to use upon login (using $HOME/.ssh/rc). Then your .bashrc or equivalent would have something like:

export SSH_AUTH_SOCK=$HOME/ssh_auth_sock

which ensures that all of your shells always point to the same symlink for the socket, which is in turn always kept up to date. More on that here.

This is super awesome. It makes your virtual terminals work forever. Except, when you are about to leave work and want to launch an automated script which will log into intranet boxes using your key, since you left work, you are no longer connected, meaning no ssh socket to use:(

The answer: ssh-agent

So we need to run a local agent (ssh-agent) to create a socket on the box that will stay no matter if we are logged in or not. This is where it gets tricky because you still have a forwarded socket created upon login which our current implementation tells our tmux session to use by default. Meaning even when you do setup ssh-agent, those changes won’t stick.

So what I needed was a bash script that can do this “registration” for me and manage it depending on the situation. The rules for this include ensuring that when I do create a local agent socket, that is used by default if available. If it is not available, we use the forwarded socket.

Here are the added lines to various files to make this all work (read the comments for info on what stuffs is doing):

To my .zshrc file (works for .bashrc as well)…note you will need to change the location of where you want the symlink to live

#Some fun ssh auth socket stuff #setting our actual variable to pnt to symlink which is what we maintain depending on socket we are using export SSH_AUTH_SOCK=$HOME/ssh_auth_sock

To my $HOME/.ssh/rc file (ran at ssh login, note your X forwarding won’t work using this…not sure how to fix that yet..)..You will also need to change the paths.

#!/bin/bash if pgrep -u $LOGNAME ssh-agent > /dev/null then #Well looks like ssh-agent is already running meaning the symlink is already pointing to the already created local socket…good to go! #Setting our backup symlink that always should point to our current remotely forwarded socket from this login ln -sf $SSH_AUTH_SOCK ~/ssh_auth_sock_rem exit else #as far as we know, ssh-agent is not running. #lets go ahead and set this up if test “$SSH_AUTH_SOCK” ; then ln -sf $SSH_AUTH_SOCK ~/ssh_auth_sock ln -sf $SSH_AUTH_SOCK ~/ssh_auth_sock_rem fi fi

And finally, my “registration” script which helps set up all this ssh-agent stuff…You will need to change a few variables up top depending on your paths…

#!/bin/bash #set some vars sshkey=“$HOME/.ssh/id_rsa.work” #the main symlink location your shell variable always points to auth_sock_loc=“$HOME/ssh_auth_sock” #the location where your forwarded socket is auth_sock_rem_loc=“$HOME/ssh_auth_sock_rem” if [ “$1” == “help” -o “$1” == “-h” -o “$1” == “?” ] then echo “This will function will launch ssh-agent and ssh-add to register your private key locally on this server for continued usage when your not logged in, and when you log back in:)” echo “running just \“reg\” will register you making sure to first check if you are already registered…running \“reg kill\” will kill any current ssh-agent processes in event of a problem/error/corruption” exit fi if [ “$1” == “kill” ] then #Maybe theres a problem…this kills all ssh-agents running as youre user echo “Killing any running agents…” ps -u $LOGNAME | grep ‘ssh-agent’ killall -u $LOGNAME ssh-agent #setting our symlink to pnt to our remote forwarded socket cp -P “$auth_sock_rem_loc” “$auth_sock_loc” echo “Done…exiting.” exit fi if pgrep -u $LOGNAME ssh-agent >/dev/null then echo “Yay!!! Already registered on this server ($(hostname))! Exiting…” exit else #Looks like we arent ssh-agented (for lack of a better word, or im just lazy) on this server echo “Not registered on $(hostname) (ssh-agent not already running)….starting up…” eval ssh-agent && ln -sf $SSH_AUTH_SOCK “$auth_sock_loc” && ssh-add “$sshkey” echo “Done! Your key is registered on this server. Any continuously running scripts should work successfully.” fi

You can see from my script that it supports asking for help and another argument called “kill” that will kill any ssh-agent you currently have running (in case of corruption, etc..) then set your symlink to your currently forwarded session so you can continue on working. Running just the script itself will launch ssh-agent, setup your symlink to point to the new socket, and run ssh-add with the key you want added. Note that this entire setup relies on your key being loaded into ssh-agent (use “ssh-add -l” to see it loaded). If your key is unloaded from ssh-agent, life is gonna suck. (not really, just kill it and rerun the script).

I’m actively using this and it works beautifully. I currently have an alias to launch this script when I type “reg”. You could also just set this up to run at login so your ssh-agent is always setup whenever your on the box:)

tl;dr, copy the stuff above to files in your crap and everything is magic from there.

Mario Loria is a builder of diverse infrastructure with modern workloads on both bare-metal and cloud platforms. He's traversed roles in system administration, network engineering, and DevOps. You can learn more about him here.