Index


Overview

Splunk is a great tool for managing and interrogating data from multiple data-sources. A Splunk installation includes the ability to create and distribute instances of the Splunk Universal Forwarder, a tool that can be deployed to a data-source server and used to collect data and send data through to your Splunk server.

This document details the process required to configure a Splunk Universal Forwarder to send your Statseeker log data through to a Splunk server deployed within your environment.

[top]


Prerequisites

  • A Statseeker server
  • A Splunk server, configured to received data from a Splunk Universal Forwarder
  • A Splunk Universal Forwarder for FreeBSD, match the forwarder version to the FreeBSD version of your Statseeker server
    Note: to confirm the FreeBSD version of your Statseeker server run the following on the server’s CLI:

    # freebsd-version

For details on acquiring, installing, and configuring a Splunk server see https://www.splunk.com

Note: this process was created and confirmed using:

  • Statseeker v5.6.2
  • Splunk Universal Forwarder v9.4.0 for 64bit FreeBSD v13

[top]


Configure the Splunk Forwarder

  • SSH into your Statseeker server as the statseeker user and then change to the root user
  • Install the forwarder:
    pkg install splunkforwarder-<version>-freebsd-<version>-amd64.txz



    Example:

    # su root
# pkg install splunkforwarder-9.4.0-freebsd13-amd64.txz
Note: if your Statseeker server cannot access the internet:

  • Download the Splunk Universal Forwarder from the Splunk website
  • Upload the forwarder to your Statseeker server
  • Navigate to the directory containing the uploaded package add the package to the server

    Example:

    # su root
# cd /tmp/
# pkg add splunkforwarder-9.4.0-251651b291fe-freebsd13-amd64.txz
  • If you receive a message that an updated version of the forwarder is available, and you’re sure that you have the correct forwarder for your FreeBSD version, then ignore the message and proceed with the install


  • Change directory to the binary location of the installed Splunk agent
  • Start the Splunk forwarder
Note: the forwarder will present a software license for review on first run, you can automate the acceptance of this license with the –accept-license argument

Example:

# cd /opt/splunkforwarder/bin
# ./splunk start --accept-license

If this is the first time that the package has been run it will prompt you to create an administration account.


  • Configure the Splunk forwarder to start whenever the Statseeker server is restarted

Example:

#./splunk enable boot-start


  • Configure the forwarder to send to the Splunk server

    Syntax: splunk add forward-server [splunk_server]:[port]
Note: the default port on the Splunk server to receive data from the forwarder is 9997. Additional ports can be configured from the Splunk web interface or by editing the inputs.conf, see Splunk’s documentation for details.

Example:

#./splunk add forward-server test122.statseeker.com:9997

OR

#./splunk add forward-server 10.1.44.3:9997


  • Add a monitor for each log you want to send through to Splunk

    Syntax: add monitor [source]

Example:

#./splunk add monitor /home/statseeker/base/logs/base-logd.log
#./splunk add monitor /home/statseeker/base/logs/httpd.errors
#./splunk add monitor /home/statseeker/base/logs/httpd.access
#./splunk add monitor /var/log/maillog

You can add additional parameters to make it easier to group and identify data sources, see Splunk’s documentation library ( http://docs.splunk.com/Documentation) for details.

You can view details on all currently configured datasources with:

#./splunk list monitor

[top]


Log Files

There are a range of logs available on a Statseeker server:
Statseeker specific logs:

  • /home/statseeker/base/logs/base-logd.log – Statseeker process logs
  • /home/statseeker/base/logs/dashboards.log – Statseeker dashboard logs
  • /home/statseeker/base/logs/httpd.error – Statseeker web-server errors (http)
  • /home/statseeker/base/logs/httpd.access – Statseeker web-server access logs(http)
  • /home/statseeker/base/logs/httpsd.error – Statseeker web-server errors (https)
  • /home/statseeker/base/logs/httpsd.access – Statseeker web-server access logs(https)
  • /home/statseeker/base/logs/python.log – Statseeker’s custom reporting error and log messages
  • /home/statseeker/base/logs/perl-logd.log – Statseeker legacy services error log

FreeBSD logs:

  • /var/log/maillog – mail server log
  • /var/log/cron – system-wide cron activity
  • var/log/messages – generic system activity (non-kernel boot errors, application/service errors)
  • /var/log/auth.log – server authentication logs

Sending Statseeker Timeseries and Event data via the Forwarder

You can also send any Statseeker data to Splunk (timeseries, event, configuration) via the Universal Forwarder.

This method requires that you also:

  • Write a script to output your data in a format which is suitable for the data and supported by Splunk (CSV, JSON, etc.) and configure the Forwarder to monitor the file
  • Configure an index on the Splunk server
  • Specify the target index when directing the Forwarder to monitor the file – in the Forwarder’s inputs.conf

    [monitor:///path/to/your/file.json]
index = my_custom_index
sourcetype = json

Refer to Splunk’s documentation ( http://docs.splunk.com/Documentation) for details.

Note: This method is generally not recommended (see below) and instead look to one of the alternate connection types:

Some items to be aware of if taking this approach:

  • The script will typically require libraries which are non-standard for the Statseeker server
    • The Statseeker upgrade process will flag these libraries as an issue and will remove\replace them during upgrade – we suggest not running the script from the Statseeker server
    • Updating libraries that Statseeker relies upon (outside of the Statseeker server update process) is very strongly discouraged
  • The Statseeker backup process will ignore the Forwarder, and may ignore associated scripts and output data

[top]