Index


Overview

Splunk is a great tool for managing and interrogating data from multiple data-sources. This document illustrates the process required to retrieve data from Statseeker’s REST API and send that data to a Splunk server via Splunk’s HTTP Event Collector (HEC).

The HEC:

  • Is a Splunk endpoint that can receive Statseeker event or metrics data via HTTP\HTTPS using token-based authentication
  • Accepts raw logs, JSON events, or key-value pairs, allowing easy field extractions
  • Is useful for real-time analytics as the HEC processes data immediately upon receipt
  • Is great for sending data to Splunk on an ad-hoc basis

In this process we:

  • Create a Splunk Index to store the Statseeker data
  • Configure an HEC token for authorizing with the HEC
  • Present an outline for a Python script which retrieves data from the Statseeker API and sends this to the HEC
    Note: the script presented in this document:

    • Is for illustrative purposes only and IS NOT SUITABLE FOR ANY PRODUCTION ENVIRONMENT
    • Employs logic for processing data specific to the example Statseeker API query used – this logic is not applicable to data returned from an alternate query

[top]


Prerequisites and Requirements

  • A Statseeker server
  • A Splunk server

For details on acquiring, installing, and configuring a Splunk server see https://www.splunk.com.

Note: this process was created and confirmed using:

  • Statseeker v5.6.2
  • Splunk Enterprise Server v9.4.0

Some references and screenshots may not align with other versions, but the process will be largely the same. The configuration of Splunk elements will be described using the Splunk server’s web interface. The configuration can also be done via the CLI, refer to Splunk’s documentation ( http://docs.splunk.com/Documentation) for details.

[top]

Configure the Splunk Index

The Splunk Index houses the Statseeker data that will be ingested by Splunk and will be referenced in the HEC configuration.

  • Access the Splunk server’s web GUI
  • Select Settings > Data > Indexes
  • Specify the index Name – this is the reference that will be used in the HEC configuration
  • Set the Index Data Type – any number of metrics or events can be sent to the index, but Splunk does not support a mix of event and metrics data in the one index
  • Configure other elements of the index as required by your environment and needs
    Note: if the configuring a Metrics index and providing timestamps with your Statseeker data, set timestamp resolution to seconds
  • Click Save


[top]

Configure the HEC

The HEC configuration:

  • Provides a token which is referenced by your script to authenticate with the Splunk server
  • Specifies the Splunk Index that should receive the data

To configure the HEC:

  • Select Settings > Data Inputs > HTTP Event Collector > New Token
  • Specify a Name and click Next
  • (Optional) Set the Source Type – this is a meta-data label which can assist efficient data processing within the index
  • Select the previously created Index
    Note: multiple indexes can be selected to receive the data. If multiple are selected, be sure to set the Default Index.
  • Click Review

  • Review the configuration, adjust as necessary and click Submit

The HEC token will be displayed.

If this is the first HEC token configured on the server, then you will need to enable HEC tokens:

  • Select Settings > Data Inputs > HTTP Event Collector > Global Settings
  • Set All Tokens = Enabled

[top]

Sending the Data

We will be using a Python script to:

  • Retrieve data from Statseeker’s REST API
  • Format the data for Splunk
  • Transmit the data to the HEC
Note: the script presented:

  • Is for illustrative purposes only and IS NOT SUITABLE FOR ANY PRODUCTION ENVIRONMENT
  • The example (see the Example tab below) employs logic for processing data specific to the example Statseeker API query used – this logic is not applicable to data returned from an alternate query

[top]

Script Requirements

Updating the script for use within your environment will require familiarity with Python.

To make use of the Python script:

  • Copy the script to a location within your environment that can access both the Statseeker server and the Splunk server. ** This script should not reside on your Statseeker server **
  • Ensure that all libraries listed in the import block are available to the local Python environment
  • Update the following variables with values specific to your environment and requirements:
    • statseeker_server – replace with your Statseeker server IP
    • statseeker_api_user – replace with your Statseeker API user account name
    • statseeker_api_pword – replace with your Statseeker API user account password
    • query – replace with your API query
    • splunk_server – replace with your Splunk server IP
    • hec_token – replace with the HEC token specific to your HEC configuration
    • hec_port – Splunk’s HEC port defaults to 8088, only update this value if you updated the port in the HEC Global Settings
  • Add logic to format the data returned by the query for consumption by Splunk (the Example tab contains logic specific to the data returned by the example query)
Note: the HEC Global Settings offers the option to Enable SSL to encrypt communications with the HEC. Our example script does not employ SSL in communication with either the Statseeker or Splunk servers.
Python ScriptExample

#!/usr/bin/python
import requests
import json

def do_request(statseeker_server, query, statseeker_api_user, statseeker_api_pword):
	headers = {'Accept': 'application/json', 'Content-Type': 'application/x-www-form-urlencoded'}
	url = f'https://{statseeker_server}/ss-auth'
	authData = {'user': statseeker_api_user, 'password': statseeker_api_pword}

	resp = requests.post(url, headers=headers, data=authData, verify=False)

	print(f'Authentication Response Status Code: {resp.status_code}')  # Print status code

	headers['Content-Type'] = 'application/json'
	url = f'https://{statseeker_server}/{query}'

	if resp.status_code == 200:
		myToken = resp.json()
		headers['Authorization'] = f'Bearer {myToken["access_token"]}'
		resp = requests.get(url, headers=headers, verify=False)

	elif resp.status_code == 401:
		print(f'Token auth failed: {resp.status_code}')
	return resp


# Statseeker Server IP
statseeker_server = '<STATSEEKER_SERVER>' # Replace with your Statseeker server IP

# credentials
statseeker_api_user = '<STATSEEKER_API_USER>' # Replace with your Statseeker API user

statseeker_api_pword = '<API_USER_PASSWORD>' # Replace with your Statseeker API user password

# API query
query = '<STATSEEKER_API_QUERY>' # Replace with your Statseeker API query

# Run the request
resp = do_request(statseeker_server, query, statseeker_api_user, statseeker_api_pword)

splunk_server = "<SPLUNK_SERVER_IP>" # Replace with your Splunk server IP
hec_token = "<HEC_TOKEN>"  # Replace with your HEC token
hec_port = "8088"  # Replace **ONLY** if you update the default HEC port on your server

hec_url = f"http://{splunk_server}:{hec_port}/services/collector/event"

if resp.status_code == 200:
	try:
		data = json.loads(resp.text)
		objects = data['data']['objects']

		my_data = []  # Initialize a list to house data

		for obj in objects:
			if 'data' in obj and obj['data']:  # Check that API response data exists and is not null
				
				# ** START query-specific data processing **
				
				# Custom logic goes here - See the Example tab for a sample query and query-specific logic

				# ** END query-specific data processing **

		if my_data:  # Check that there is data to send            

			# Prepare the message for the HEC
			headers = {'Authorization': f'Splunk {hec_token}'}
			data_to_send = json.dumps(my_data)

			print("Data to be sent:")
			print(json.dumps(my_data, indent=4))
			
			# Send data to HEC
			try:
				response = requests.post(hec_url, headers=headers, data=data_to_send, verify=False)

				if response.status_code != 200:
					print(f"Error sending to HEC: {response.status_code} {response.text}")
				else:
					print(f"Data sent to HEC successfully")

			except requests.exceptions.RequestException as e:
				print(f"Connection error: {e}")
		else:
			print("No events to send.")
	except json.JSONDecodeError as e:
		print(f"Error decoding JSON response from Statseeker: {e}")
		print(f"Raw response text: {resp.text}")
else:
	print(f'Error: {resp.status_code} {resp.reason}')
	



#!/usr/bin/python
import requests
import json

def do_request(statseeker_server, query, statseeker_api_user, statseeker_api_pword):
	headers = {'Accept': 'application/json', 'Content-Type': 'application/x-www-form-urlencoded'}
	url = f'https://{statseeker_server}/ss-auth'
	authData = {'user': statseeker_api_user, 'password': statseeker_api_pword}

	resp = requests.post(url, headers=headers, data=authData, verify=False)

	print(f'Authentication Response Status Code: {resp.status_code}')  # Print status code

	headers['Content-Type'] = 'application/json'
	url = f'https://{statseeker_server}/{query}'

	if resp.status_code == 200:
		myToken = resp.json()
		headers['Authorization'] = f'Bearer {myToken["access_token"]}'
		resp = requests.get(url, headers=headers, verify=False)

	elif resp.status_code == 401:
		print(f'Token auth failed: {resp.status_code}')
	return resp


# Statseeker Server IP
statseeker_server = '100.2.22.22' # Replace with your Statseeker server IP

# credentials
statseeker_api_user = 'api_user' # Replace with your Statseeker API user

statseeker_api_pword = 'api_user_password' # Replace with your Statseeker API user password

# API query
query = '/api/v2.1/cdt_device/?fields=name,ping_rtt&formats=vals&value_time=start&groups=Routers&timefilter=range=now -4h to now&links=none&limit=0' # Replace with your Statseeker API query

# Run the request
resp = do_request(statseeker_server, query, statseeker_api_user, statseeker_api_pword)

splunk_server = "100.2.22.88" # Replace with your Splunk server IP
hec_token = "123abc456def-1234-5678-abcd-123abc456def"  # Replace with your HEC token
hec_port = "8088"  # Replace **ONLY** if you update the default HEC port on your server

hec_url = f"http://{splunk_server}:{hec_port}/services/collector/event"

if resp.status_code == 200:
	try:
		data = json.loads(resp.text)
		objects = data['data']['objects']

		my_data = []  # Initialize a list to house data

		for obj in objects:
			if 'data' in obj and obj['data']:  # Check that API response data exists and is not null
				
				# ** START query-specific data processing **
				# The content of this loop is specific to our example API query 
				for device in obj['data']:

					device_name = device['name']
					ping_rtt_data = device['ping_rtt']['vals']

					for timestamp, rtt in ping_rtt_data:
						payload = {
							"event": "metric",
							"time": timestamp,
							"source": "statseeker",
							"sourcetype": "statseeker_ping",
							"host": "statseeker",
							"fields": {
								"metric_name:ping_rtt": rtt,
								"device_name": device_name
							}
						}
						my_data.append(payload) 
				# ** END query-specific data processing **

		if my_data:  # Check that there is data to send            

			# Prepare the message for the HEC
			headers = {'Authorization': f'Splunk {hec_token}'}
			data_to_send = json.dumps(my_data)

			print("Data to be sent:")
			print(json.dumps(my_data, indent=4))
			
			# Send data to HEC
			try:
				response = requests.post(hec_url, headers=headers, data=data_to_send, verify=False)

				if response.status_code != 200:
					print(f"Error sending to HEC: {response.status_code} {response.text}")
				else:
					print(f"Data sent to HEC successfully")

			except requests.exceptions.RequestException as e:
				print(f"Connection error: {e}")
		else:
			print("No events to send.")
	except json.JSONDecodeError as e:
		print(f"Error decoding JSON response from Statseeker: {e}")
		print(f"Raw response text: {resp.text}")
else:
	print(f'Error: {resp.status_code} {resp.reason}')


  • Update the script as needed
  • Search the index to return your Statseeker data


[top]