Index
- Overview
- Prerequisites
- Configure the Splunk Forwarder
- Log Files
- Sending Statseeker Timeseries and Event data via the Forwarder
Overview
Splunk is a great tool for managing and interrogating data from multiple data-sources. A Splunk installation includes the ability to create and distribute instances of the Splunk Universal Forwarder, a tool that can be deployed to a data-source server and used to collect data and send data through to your Splunk server.
This document details the process required to configure a Splunk Universal Forwarder to send your Statseeker log data through to a Splunk server deployed within your environment.
Prerequisites
- A Statseeker server
- A Splunk server, configured to received data from a Splunk Universal Forwarder
- A Splunk Universal Forwarder for FreeBSD, match the forwarder version to the FreeBSD version of your Statseeker server
Note: to confirm the FreeBSD version of your Statseeker server run the following on the server’s CLI:
# freebsd-version
For details on acquiring, installing, and configuring a Splunk server see https://www.splunk.com
- Statseeker v5.6.2
- Splunk Universal Forwarder v9.4.0 for 64bit FreeBSD v13
Configure the Splunk Forwarder
- SSH into your Statseeker server as the statseeker user and then change to the root user
- Install the forwarder:
pkg install splunkforwarder-<version>-freebsd-<version>-amd64.txz
Example:# su root # pkg install splunkforwarder-9.4.0-freebsd13-amd64.txz
- Download the Splunk Universal Forwarder from the Splunk website
- Upload the forwarder to your Statseeker server
- Navigate to the directory containing the uploaded package add the package to the server
Example:# su root # cd /tmp/ # pkg add splunkforwarder-9.4.0-251651b291fe-freebsd13-amd64.txz
- If you receive a message that an updated version of the forwarder is available, and you’re sure that you have the correct forwarder for your FreeBSD version, then ignore the message and proceed with the install
- Change directory to the binary location of the installed Splunk agent
- Start the Splunk forwarder
Example:
# cd /opt/splunkforwarder/bin # ./splunk start --accept-license
If this is the first time that the package has been run it will prompt you to create an administration account.
- Configure the Splunk forwarder to start whenever the Statseeker server is restarted
Example:
#./splunk enable boot-start
- Configure the forwarder to send to the Splunk server
Syntax: splunk add forward-server [splunk_server]:[port]
Example:
#./splunk add forward-server test122.statseeker.com:9997 OR #./splunk add forward-server 10.1.44.3:9997
- Add a monitor for each log you want to send through to Splunk
Syntax: add monitor [source]
Example:
#./splunk add monitor /home/statseeker/base/logs/base-logd.log #./splunk add monitor /home/statseeker/base/logs/httpd.errors #./splunk add monitor /home/statseeker/base/logs/httpd.access #./splunk add monitor /var/log/maillog
You can add additional parameters to make it easier to group and identify data sources, see Splunk’s documentation library ( http://docs.splunk.com/Documentation) for details.
You can view details on all currently configured datasources with:
#./splunk list monitor
Log Files
There are a range of logs available on a Statseeker server:
Statseeker specific logs:
- /home/statseeker/base/logs/base-logd.log – Statseeker process logs
- /home/statseeker/base/logs/dashboards.log – Statseeker dashboard logs
- /home/statseeker/base/logs/httpd.error – Statseeker web-server errors (http)
- /home/statseeker/base/logs/httpd.access – Statseeker web-server access logs(http)
- /home/statseeker/base/logs/httpsd.error – Statseeker web-server errors (https)
- /home/statseeker/base/logs/httpsd.access – Statseeker web-server access logs(https)
- /home/statseeker/base/logs/python.log – Statseeker’s custom reporting error and log messages
- /home/statseeker/base/logs/perl-logd.log – Statseeker legacy services error log
FreeBSD logs:
- /var/log/maillog – mail server log
- /var/log/cron – system-wide cron activity
- var/log/messages – generic system activity (non-kernel boot errors, application/service errors)
- /var/log/auth.log – server authentication logs
Sending Statseeker Timeseries and Event data via the Forwarder
You can also send any Statseeker data to Splunk (timeseries, event, configuration) via the Universal Forwarder.
This method requires that you also:
- Write a script to output your data in a format which is suitable for the data and supported by Splunk (CSV, JSON, etc.) and configure the Forwarder to monitor the file
- Configure an index on the Splunk server
- Specify the target index when directing the Forwarder to monitor the file – in the Forwarder’s inputs.conf
[monitor:///path/to/your/file.json] index = my_custom_index sourcetype = json
Refer to Splunk’s documentation ( http://docs.splunk.com/Documentation) for details.
Some items to be aware of if taking this approach:
- The script will typically require libraries which are non-standard for the Statseeker server
- The Statseeker upgrade process will flag these libraries as an issue and will remove\replace them during upgrade – we suggest not running the script from the Statseeker server
- Updating libraries that Statseeker relies upon (outside of the Statseeker server update process) is very strongly discouraged
- The Statseeker backup process will ignore the Forwarder, and may ignore associated scripts and output data