In the course of maintaining a complex infrastructure, its common to have multiple ssh sessions open on multiple servers. This can make it very easy to accidentally push changes to the wrong server. This week, we’ll be sharing a best practice we use at SIPfish to avoid making this mistake.
Red means danger!!!
The problem is that one ssh session looks much like any other, whether its a production web server or a local staging server. Our solution to this is to hijack the ssh command and compare the destination to a list of production servers. If there’s a match, the screen color is changed on the way to the real ssh.
The list of production boxes is kept in ~/.prod_boxes. It can contain a mix of ip addresses and domains.
10.100.10.1 10.100.10.2 sipfish.com www.sipfish.com
We can catch calls to ssh by putting this line in ~/.bashrc or ~/.profile if you’re on a Mac:
alias ssh="~/bin/safe_ssh $1"
Here’s the contents of ~/bin/safe_ssh:
#!/bin/bash ip=`echo $1 | cut -d"@" -f2` match=`cat ~/.prod_boxes | grep $ip | wc -l` if [ $match -gt 0 ] then tput setab 1 tput setaf 14 clear fi
The only problem with this is that running a command on the remote server that adjusts the screen or text color, such as vi, will clobber these changes. Most of our developers work on a Mac, which allows them to use AppleScript to change the colors of their Terminal tab. Here’s the Mac version of ~/bin/safe_ssh:
#!/bin/bash ip=`echo $1 | cut -d"@" -f2` match=`cat ~/.prod_boxes | grep $ip | wc -l` if [ $match -gt 0 ] then osascript ~/bin/prod_theme.scpt fi
and here’s ~/bin/prod_theme.scpt
tell application "Terminal" to set current settings of selected tab of window 1 to (first settings set whose name is "Red Sands")
Best practices aren’t a guarantee against stupidity
You can, of course, still become enured to a red window and make changes without thinking. In order for this to work well, you have to be consistent about closing your terminal windows when you’re done making production changes. Don’t let yourself log out and start working locally even though you still have that red screen. Also, you need to make sure that your list of production servers is accurate and up to date. Or better yet, use a configuration management tool to administer production boxes. Chef and Capistrano are the two that we use.