« Installing nrpe on OS X | Main | Installing Splunk on Ubuntu »

Monday, May 16, 2011

Timelox and TheHand

The Problem

Since 2004 we've seen an increase in the number of brute force attempts to break into machines via ssh. These attempts appear to use the same password across different accounts, and thus do not trigger many of the normal methods of detecting brute force, such as pam_tally, nor do they result in the typical slowdowns associated with repeated attempts at a single ID.

Murray Anderegg turned me onto timelox, written by a fellow named Elad Efrat (aka Brian at ethernet.org). This code tracks failed logins by ip number, but doesn't care about which ids were hit, and when a threshold is reached, the offending ip number is put into a deny rule for a firewall.

The one thing I didn't like about it so much was that the code called the firewall explicitly from within ssh, so you'd have to change the patch code for different oses. And Murray wanted to make some changes in the source code to meet his needs with iptables under linux. So what we did was modify Elad's patch code and abstracted things a bit so that instead of calling a firewall directly, you can set a variable in the sshd_config file specifying a script that's called when the threshold is reached. I _think_ it's relatively safe, but I'm not that good at this kind of thing, so your mileage may vary. I can't be responsible if you use this code and something bad happens, whether that is someone hacking your machine, locusts attacking your rose bushes, or not-quite-flying cattle impacting your home in an adverse manner.

This particular doc is primarily oriented to OS X, but the timelox code works fine on Redhat and probably pretty much an flavor of linux, and there's an example script for iptables provided.

Why do it this way?

Well, partly to keep things portable, and partly since I think calling an external script is a more attractive option, since that keeps the patching of ssh to a minimum, since the same patch can be used for pretty much any operating system.

Also, although this example uses a timelox patch in ssh to call our scripts, one can use any manner of software as a trigger. For example, one could make a honeypot program to watch a socket and mimic ftp or other services, and call this script when anything odd is seen. Or one could call a completely different script to fire a report, trigger a firewall at the head of a network, etc.

What's here

Please note that the patches and scripts were updated around the time this page was last modified (see below for the date), the changes are pretty significant including changes in path names and script names in the default installation. 

All of the files are located here. If you're in a hurry, just go for the latest tgz of the installer , which includes openssh-5.8p1.

Installation Instructions

In a basic installation there are four steps:

  1. Patch (or download a patched copy of) openssh with the timelox code, and install it
  2. Modify the sshd_config file with the lines supporting timelox
  3. Copy TheHand.sh to the location where timelox can touch it
  4. Test
I've written an installation script for OS X, and this should simplify things greatly. The installer does not include a firewall, just the openssh code and the timelox additions. To use this, first open a terminal window, then run the following commands:

curl --raw -k -o openssh_timelox_installer-current.tgz \
tar -xvzf openssh_timelox_installer-current.tgz
Next cd into the result folder, as of today that would be openssh_5.8p1.installation Then you can run the installer, the usual way would be
sudo ./install.sh -a 
Other flags you can use:
  • install.sh -c will check for necessary files, unpack the source, patch it, and then configure the software.
  • install -m will run a make clean and a make of the software. You can use this make a master if you have to push out to a bunch of similar machines.
  • install -i will do an installation, but not the configure and make phases. Use this if you're installing to multiple systems that have the same OS/processor type.
The script will warn you if you do not have a firewall script in /Library/StartupItems/Firewall, but in my initial testing timelox works fine with the default firewall setting in OS X 10.5 so long as the firewall is on.

The script puts all of the ssh code in /usr/local. It also modifies the /usr/libexec/sshd-keygen-wrapper file to use the new version of openssh. It makes backup copies of the files, but the key file is /usr/libexec/sshd-keygen-wrapper--if you want to go back to the OS X version of sshd, just copy the backup version of this file back to the original location, eg:
sudo cp /usr/libexec/sshd-keygen-wrapper-091105-171749 /usr/libexec/sshd-keygen-wrapper
By default, the-hand.sh speaks a warning aloud, if you don't like this behavior, you can comment out the line. Run:
sudo vi /usr/local/libexec/the-hand.sh
and then go to the bottom of the file and find this section:
# And because it's a mac, and we can, speak the action
osascript -e "say \"Warning, the hand has blocked ${1} in i p f w with rule ${block_new_number} \"";
If you put a pound sign in front of osascript, that will suppress the audible warning, eg:
# And because it's a mac, and we can, speak the action
#osascript -e "say \"Warning, the hand has blocked ${1} in i p f w with rule ${block_new_number} \"";
For more detail on manual building of the code, you can use the information below.


  • Scripts as examples
    • the-hand.sh_ipfw: (if you're using 10.4 or earlier you must  use a custom ipfw firewall for this script to work, that's what the sample ipfw firewall listed above is for! The default firewall provided by OS X 10.5 does seem to work, since it doesn't use ipfw to set rules for inbound connections.
    • ipfw_example: An example of an ipfw firewall that intersects nicely with the ipfw version of TheHand. For more info on ipfw, see http://www.ibiblio.org/macsupport/ipfw/
    • TheHand.sh_iptables: A second older example script for linux/iptables
  • Also, do note that a particular version of openssh may not be ideal for your needs, so what you may wish to do is find out which version of openssh your distribution runs, and adapt the patch code to the source for that version. So test it throughly before deploying, and if you run into problems, you might consider adapting the patch to whatever version of openssh your system uses. Adapting the patch is pretty easy.

Editing the sshd_config file

The installer takes care of these setting for you, but here's a rundown on what's what in case you need it.

There are several variables that can be set in the sshd_config file. On OS X systems the first thing to do is enable PAM support, find the line that says "#UsePAM no" and change that to:

UsePAM yes

You also need to check out the timelox settings. Here's a list of timelox variables:

  • UseTimelox enables it,
  • TimeloxTimeFrame sets the window in seconds for failed logins
  • TimeloxMaxBadLogins sets the number of failed logins needed the timeframe to trigger timelox
  • TimeloxCall is the full path to whatever script or program you want to call
  • when the failed login threshold is reached
  • TimeloxPath is the full path to timelox's working files
  • TimeloxIPTable is just a string you can use to specify which chain in an iptables
  • firewall you want to insert blocks
  • TimeloxRuleno lets you set what rule number you want to use in a chain
  • TimeloxLogLevel lets you set a log level
  • TimeloxLabel is just an indentifier in case your script has multiple triggers.

All of these are just string or integers, so you can use them for other purposes if you like.

In this example, the number of bad logins and the time frame are both set relatively high, which is what I think makes sense for a workstation. These lines can go anywhere, so just scroll to the bottom of the file and add them there. Here are reasonable starting values

UseTimelox yes
TimeloxTimeFrame 7200
TimeloxMaxBadLogins 3
TimeloxCall /usr/local/libexec/TheHand.sh
TimeloxPath /usr/local/etc/timelox/
TimeloxIPTable INPUT
TimeloxRuleno 1
TimeloxLogLevel authpriv.info
TimeloxLabel sshd_timelox

While you're here you may want to make some other changes. Adding a banner is a good idea, if only to help you with your testing. Find the banner line and uncomment it: 

Banner /usr/local/etc/ssh/sshd_banner

TheHand script

In the example above, timelox is calling a script named TheHand.sh. This script is a bash shell script that takes variables, such the ip number and a text string used to indicate the source of the block. There are two sample scripts provided, one TheHand.sh_ipfw for Mac OS X, and TheHand.sh_iptables for redhat. Both use pretty much the same logic. The example for redhat is set up in a brutal fashion to insert rules at the beginning of the INPUT chain. This method works, but it's not elegant. The ipfw version assumes that you are using a custom firewall, and inserts rules starting at 5000. The way to use this is to set up ipfw so that local host rules are in the lower number ranges, and then put in services acceptance rules starting high (in my case, 25000). That way the deny rule inserted by timelox stays clear of your rules, and the denys occur before the service acceptance rules.

TheHand.sh does some basic sanity checks on the ip number to see that it's valid (currently only ipv4 addresses work) and then inserts a rule denying all traffic inbound from that ip address into the firewall. TheHand.sh also supports logging. There are comments throughout the script that explain pretty well what it does, and it's not very complicated. You should read through the file to make sure you understand (at least roughly) what it's doing. 


Ok, now that the bits should all be in place, time for some testing. To test our sshd installation, we'll start it with some flags in dbug mode on a non-standard port. 

sudo /usr/local/sbin/sshd -p 2222 -d

You'll see something like this:

[Avatar:/usr/local/bin] hays% sudo /usr/local/sbin/sshd -p 2222 -d
debug1: sshd version OpenSSH_4.0p1
debug1: private host key: #0 type 0 RSA1
debug1: read PEM private key done: type RSA
debug1: private host key: #1 type 1 RSA
debug1: read PEM private key done: type DSA
debug1: private host key: #2 type 2 DSA
debug1: rexec_argv[0]='/usr/local/sbin/sshd'
debug1: rexec_argv[1]='-p'
debug1: rexec_argv[2]='2222'
debug1: rexec_argv[3]='-d'
debug1: Bind to port 2222 on ::.
Server listening on :: port 2222.
debug1: Bind to port 2222 on
Server listening on port 2222.
Generating 768 bit RSA key.
RSA key generation complete. 

Now, in a different terminal window, try logging into localhost on port 2222, with:

ssh -p 2222

You should see some debugging code fly by in the window in which you started sshd, and something like this in the window you're logging in from:

[Avatar:~] hays% ssh -p 2222
* *
* Warning! Unathorized Logins Prohibited *
* *

Notice the banner--since we set this in the /usr/local/etc/ssh/sshd_config, we know we're using the right config file (and not the default mac /etc/ssh_config).  Give it your password and see if you can login. If you succeed, you've got a working sshd. Type exit, and you'll leave that ssh session and fall back into your terminal window. If you look at the other window, where you started sshd, you'll should have the prompt back. This method of running sshd in debug mode only sets up a single session, so to run another test, you have to restart it (but that's easy, just use the up arrow key to back up one in the cammand history and then hit return).

Now, restart the sshd login into a remote machine, and then try to ssh back to the machine with the new sshd installed, using the same port. This time, munge your password. What should happen is that you get three attempts, and the fourth will appear to lockup (if this worked, then when you try the fourth time, your machine with timelox will insert a firewall rule blocking all traffic from the remote machine). That block will stay in place until you reload your firewall or reboot.  


If it doesn't work the way you expect, use tail in a second terminal window to follow the systemlog:

tail -f /var/log/system.log

This should give you some clues as to what's failing.

The ipfw version of TheHand.sh should also speak aloud when triggered--if it doesn't then the path to it configured may be wrong, or there may be a permissions error. You can also run this script manually, eg:

sudo /usr/local/libexec/TheHand.sh blah blah blah 
Posted by bil at 2:34 PM
Categories: My Software, Work