Friday, November 30, 2012
Installing Splunk on Ubuntu
Recently, I've set up a splunk server to do remote logging. Splunk is easy to install, pretty easy to use with a very powerful searching system. They also allow 500 megs of logging per day in their free version (after that it gets expensive). But I can't really recommend the free level, since they removed the authentication, which makes making the service secure very difficult. I think it would be find to run at home behind your NAT, but not with a public IP address. So I'm looking into syslog-ng and some other options. But I figured I'd post this in case it might help someone.
Get the splunk installer, in this case we'll use the .deb version since we're installing to Unbuntu. To do this, you'll need to sign up at splunk.com. I don't think any of this is FOSS, so I can't post it here.
Splunk likes to live in /opt, so make a /opt if there's not one,
sudo mkdir /opt;
Now, install the .deb file, in this example, the .deb file is in my home directory:
dpkg -i ~/splunk-4.3.1-119532-linux-2.6-amd64.deb
This shouldn't take long. At the end, you'll be prompted for how to start splunk.
/opt/splunk/bin/splunk start --accept-license
Let it run until you get back to a prompt. Further configuration is done via the web interface. Start your browser, login as admin and use changeme, then change to a better password. You can do this because right now your in a trial period--once that expires, you won't have to authenticate to reach the splunk server--and neither will anyone else.
The web interface will let you control most settings, if you click on Manager in the upper right menu, you'll be able to adjust most of the settings.
Since the main purpose of a splunk server is to accept logs from remote machines, we'll want to set up a couple of receivers, one for encrypted connections and one for non-encrypted. The latter we'll use primarily for testing, as we wouldn't want to pass logs for important servers in clear text. Under:
Manager, Data, Forwarding and receiving
create a listening on port 9997 and one on 9443
Configure Receiving, listen on 9997
You can also configure the listeners via command line by editing:
One of the trickier bits was setting up certificates. You can use the certs provided by splunk as is, and for many, that's fine. But I went ahead and generated new ones with different password from the default as a security precaution. The basic logic is you make self-signed certs for the forwarder (the software forwarding the logs to splunk) to use in order to connect over SSL. The passwords go into the inputs.conf (on the server) and outputs.conf (on the forwarders), and they get hashed the next time the server starts.
First, make a place to store your certs out of the way of the default certs.
sudo mkdir -p ./certs;
Then run a splunk script to make the certs:
/opt/splunk/bin/genSignedServerCert.sh -d /opt/splunk/etc/certs -n server -c YOURSERVER.NAME.HERE -p
Use the Fully Qualified Domain Name (FQDN) for the server. The script will prompt for a password for the cert, and also for more information. Enter the server's FQDN for common name, all other entries are optional. This will make a server.pem file you can use in the inputs.conf file for the server.
Now, run that script again to make a cert for the forwarder.
/opt/splunk/bin/genSignedServerCert.sh -d ./ -n forwarder -p
Use the same or different password, and all other settings are optional. This will create a file named forwarder.pem. To use these, you'll make a certs folder in the forwarder's directory, and copy both the forwarder.pem and cacert.pem there.
Finally, to enable the server to run at boot:
sudo /opt/splunk/bin/splunk enable boot-start
That's it, best of luck!