« December 2012 | Main | May 2011 »

Friday, November 30, 2012

Installing Splunk on Ubuntu

Recently, I've set up a splunk server to do remote logging. Splunk is easy to install, pretty easy to use with a very powerful searching system. They also allow 500 megs of logging per day in their free version (after that it gets expensive). But I can't really recommend the free level, since they removed the authentication, which makes making the service secure very difficult. I think it would be find to run at home behind your NAT, but not with a public IP address. So I'm looking into syslog-ng and some other options. But I figured I'd post this in case it might help someone.

Installing splunk

Get the splunk installer, in this case we'll use the .deb version since we're installing to Unbuntu. To do this, you'll need to sign up at splunk.com. I don't think any of this is FOSS, so I can't post it here.

Splunk likes to live in /opt, so make a /opt if there's not one,

sudo mkdir /opt;

Now, install the .deb file, in this example, the .deb file is in my home directory:

dpkg -i ~/splunk-4.3.1-119532-linux-2.6-amd64.deb

This shouldn't take long. At the end, you'll be prompted for how to start splunk.

Start splunk

/opt/splunk/bin/splunk start --accept-license

Let it run until you get back to a prompt. Further configuration is done via the web interface. Start your browser, login as admin and use changeme, then change to a better password. You can do this because right now your in a trial period--once that expires, you won't have to authenticate to reach the splunk server--and neither will anyone else.

The web interface will let you control most settings, if you click on Manager in the upper right menu, you'll be able to adjust most of the settings.

Since the main purpose of a splunk server is to accept logs from remote machines, we'll want to set up a couple of receivers, one for encrypted connections and one for non-encrypted. The latter we'll use primarily for testing, as we wouldn't want to pass logs for important servers in clear text. Under:

Manager, Data, Forwarding and receiving

create a listening on port 9997 and one on 9443

Configure Receiving, listen on 9997

You can also configure the listeners via command line by editing:


Securing connections

One of the trickier bits was setting up certificates. You can use the certs provided by splunk as is, and for many, that's fine. But I went ahead and generated new ones with different password from the default as a security precaution. The basic logic is you make self-signed certs for the forwarder (the software forwarding the logs to splunk) to use in order to connect over SSL. The passwords go into the inputs.conf (on the server) and outputs.conf (on the forwarders), and they get hashed the next time the server starts.

This is all based on http://splunk-base.splunk.com/answers/7164/how-do-i-set-up-ssl-forwarding-with-new-self-signed-certificates-and-authentication


First, make a place to store your certs out of the way of the default certs.

cd /opt/splunk/etc/

sudo mkdir -p ./certs;

Then run a splunk script to make the certs:

/opt/splunk/bin/genSignedServerCert.sh -d /opt/splunk/etc/certs -n server -c YOURSERVER.NAME.HERE -p

Use the Fully Qualified Domain Name (FQDN) for the server. The script will prompt for a password for the cert, and also for more information. Enter the server's FQDN for common name, all other entries are optional. This will make a server.pem file you can use in the inputs.conf file for the server.

Now, run that script again to make a cert for the forwarder.

/opt/splunk/bin/genSignedServerCert.sh -d ./ -n forwarder -p

Use the same or different password, and all other settings are optional. This will create a file named forwarder.pem. To use these, you'll make a certs folder in the forwarder's directory, and copy both the forwarder.pem and cacert.pem there.

Finally, to enable the server to run at boot:

sudo /opt/splunk/bin/splunk enable boot-start

That's it, best of luck!

Posted by bil at 7:02 PM
Categories: Other Software, Work