Index


Overview

Statseeker can be used as a comprehensive proactive, and reactive, alerting solution. Alerts can be triggered for:

  • Threshold Events - thresholds can be configured against any timeseries metric monitored by Statseeker. By configuring thresholds Statseeker can send alerts when your network is experiencing unusual or unwanted activity such as excessive temperatures, memory\CPU load, or a device\link is nearing capacity. For more details, see Threshold Configuration.
  • Device Events - device level events, such as ping state changes, on any devices monitored by Statseeker
  • Interface Events - interface level events, such as ifOperStatus state changes, on any interfaces monitored by Statseeker
  • Syslog Events - if your network contains devices that have been configured to output messages to a remote syslog server, then these devices can send those messages to Statseeker. Statseeker can review these logs and use the content to trigger alerts relating to those events.
  • SNMP Trap Events - if your network contains devices that have been configured to output SNMP trap messages to Statseeker, then Statseeker can review these messages and trigger alerts based on their content.
Note: the Admin Tool contains settings for how long to store event data records, for details see Syslog, Trap and Event Data Historical Records.

Alerting is achieved through:

  • Targeted Emails: a range of email templates and the ability to customize content makes it simple to provide the level of detail needed to respond to an issue efficiently. You will also need to configure Statseeker's mail server to send email alerts, see Email Configuration for more information.
  • Syslog Messages: Statseeker events can be used to trigger an alert which sends a syslog message to a collector\aggregator
  • Custom Script: run a custom shell script in response to a specific alert trigger

Once an alert configuration has been enabled, it will be triggered by every subsequently recorded event satisfying the requirements for the alert. In some instances, such as during scheduled maintenance windows, you may wish to suppress some alerts. This can be achieved by disabling those alerts, see Disabling Alerts for more. When a more targeted approach is needed, Event Management rules can be defined to discard a specific type of event from a specified device (or group of devices). For more information, see Event Management.

[top]


Configuring Alerts

Note:

Statseeker's Alerting allows for great variability in alerting configurations. The available configuration options vary with the selected Template, Alert Type, and Data Type. A complete walk-through of these options and their application is also available in video format.


To configure a new alert:

  • Select Administration Tool > Alerting / Event Management > Alerting

This will display a list of currently configured alerts.

  • Click Add

This will display the New Alert screen.

Field Description
Template Alert template to use, for details on templates see Alert Templates

Note: the available templates can be edited from within the Admin Tool:

  • Select Admin Tool > Network Discovery - Advanced Options > Advanced Options and click File Edit Mode
  • Add a new item 'alerting_allowed_templates', set the values and click Save

The allowed templates can be specified as a space\tab separated list with values of:

  • 'email_simple'
  • 'email_custom'
  • 'email_advanced'
  • 'syslog'
  • 'script'

Example:

'alerting_allowed_templates' 'email_custom' 'syslog'
Name Name of the alert configuration
Status
  • Enabled - alerts are sent for this configuration
  • Disabled - the alert configuration is retained once saved, but does not generate alerts
Alert Type What will trigger the alert:

  • Event - many data types include 'state' fields. When a monitored state changes, Statseeker records an event (detailing the state change) which can be used to trigger an alert.
    Examples:
    • Ping up/down events (availability from a device's default ping poller only)
    • SNMP availability changes
    • Interface operation status (ifOperStatus) changes
    • UPS battery status
  • Syslog - devices can be configured to forward syslog records to your Statseeker server, alerts can be configured to trigger in response to the content of these messages
  • Trap - SNMP trap messages can be forwarded to your Statseeker server, alerts can be configured to trigger in response to the content of these messages
  • Threshold - thresholds can be configured against any timeseries metric monitored by Statseeker (see Threshold Configuration for details) and alerts can be configured to trigger in response to threshold breaches
  • Ping - ping state as recorded from any ping poller monitoring the device
Note: Statseeker allows for multiple ping pollers to be distributed throughout your network (see Observability Appliances for details). When multiple pollers are available, each device has a designated default poller, and can have any number of additional pollers monitoring the device. When:

  • Event Type = Device and Field = Ping State, the alert will only respond to ping state changes recorded by the default poller
  • Event Type = Ping and Field = Ping State, the alert can be configured to respond to ping state changes recorded by any specified pollers. including the default poller
Data Type Requires Alert Type = Event; Syslog; Trap
Select the data type the alert will be configured against. This is a filtered list only displaying those data types currently being monitored by your server.
Threshold Requires Alert Type = Threshold
Select the threshold the alert will be configured against
Field Requires Data Type
Select a state field associated with the selected Data Type
States Requires Field or Alert Type = Threshold
Select the states which will trigger the alert
Regex Requires Alert Type = Syslog; Trap
Specify a RegEx to filter Syslog\Trap message records by content. Receipt of a message which satisfies the filter will result in the alert being triggered.
Entity/Group Filter Only events originating from the specified device or group can trigger alerts - not all entities are presented at once, begin typing to filter the list
Time Filter Matching events which occur within the specified time period will generate alerts. For more information on generating these, see Time Filters
Time Filter Mode Determines if the alert is triggered by events in effect at the beginning of the alerting period.
E.g. You have an alert which references an 'on transition' threshold. If at the start of the alerting period (as specified by the Time Filter), the threshold is in breach.

  • Ignore all events outside time filter range - no alert is triggered until the threshold transitions once again
  • Alert if down or breached at start of time filter range - an alert is triggered at the start of the reporting period
Host Requires Template = Syslog
Specify the IP address of the target syslog collector\aggregator
Format Requires Template = Syslog
Specify the output syslog message format

  • Statseeker - legacy format, contains the following
    • Message number
    • Timestamp
    • Facility code
    • Application/Process
    • Loghost
    • Message
  • JSON - standard JSON message format, includes all of the details from the Statseeker format and offers additional fields containing configuration data from the message source
Time Zone Used in conjunction with the time filter
Enable Logging Only enable logging when troubleshooting alerting issues
Bundling Requires Template = Email
Bundling policy to be used, for details see Bundling Policies
Waiting Requires Template = Email with custom content (advanced)
Specify an alternate bundling action to be taken when device ping state events are received, for details see Bundling Policies
Upstream Devices Requires Template = Email with custom content (advanced)

Use your upstream device configurations to suppress ping/OperStatus down alerts from downstream devices when an alert exists for an upstream neighbor, see Upstream Device Configuration for details
Email To Requires Template = Email

  • An email address
  • A Statseeker user account (uses the email address associated with the account)
  • A group containing Statseeker users (uses the email addresses associated with each user account)
Email Subject Requires Template = Email

Email subject line
Mode Requires Template = Email with custom content

  • HTML - (default) HTML content formatting and presentation
  • Raw Text - simple text content
Note:

  • Raw Text: converts the content to raw text prior to sending, removing formatting where possible and stripping content\functionality that cannot be converted
  • When Mode = HTML the email contains both the HTML and raw text content, when a recipient's mail client is unable to process the HTML message, it will fall back to the Raw Text message
  • Alert configurations from earlier Statseeker versions (pre-v5.6.1) will be set to Raw Text upon server upgrade. Change these configurations to Mode=HTML to take advantage of the new functionality
  • Images can be added to HTML email alerts from the Insert menu, or embedded into your email via copy-paste
Content Requires Template = Email with custom content

Specify the alert email content. This content can include text, links, images, tables, and variables.

Note:

  • Variables - select from a list of short-codes to include variable data relating to the event\infrastructure generating the alert, see Email Variables for details
  • Preview Test Email can be used to preview the email output
Recipient Roster File Requires Template = Email with custom content (advanced)

Specify a text file detailing an email recipient roster. Each line in the roster file should contain a start date-time followed by one or more email addresses.

Recipient Roster Syntax

The format for the date-time is YYYY MM DD HH MM.

Example: 2023 02 03 17 30 - 2023, February the 3rd, at 5:30pm.

The format for the roster file is:

{date-time} {email1} {email2} ... {emailn}
{date-time2} {email1} {email2} ... {emailn}
{date-time3} {email1} {email2} ... {emailn}

** Ensure that there are no leading or trailing (blank) lines in the roster file **

Example:

2023 02 03 17 30 sysadmin1@example.com sysadmin2@example.com
2023 02 10 17 30 sysadmin3@example.com sysadmin4@example.com
2023 02 17 17 30 sysadmin1@example.com sysadmin2@example.com
2023 02 24 17 30 sysadmin3@example.com sysadmin4@example.com
2023 03 03 17 30 sysadmin1@example.com sysadmin2@example.com

Once a date-time is reached, all future alerts generated from this configuration will be addressed to the specified email addresses. When the next date-time is reached, the recipients are replaced by those specified.
Subject Date Format Requires Template = Email with custom content (advanced)

Select an alternate date-time format to be used for events times when a date-time variable is included in the email subject line
Body Date Format Requires Template = Email with custom content (advanced)

Select an alternate date-time format to be used for events times when date-time variable is included in the email body
Newlines Requires Template = Email with custom content (advanced)

Apply improved email body content layout for Windows end users, disable when email recipients are non-Windows users
Check Event Filters Runs the alert configuration against the Events database and returns recent events that would trigger the alert.
Preview Test Email Requires Template = Email

Display a preview of the email content. This is useful when adding variables to your alert email subject line or body.
Send Test Email Test mail server and alert configuration by sending a test alert email
Note: existing alerts can be cloned to speed up the process of creation. This is particularly useful when creating a number of very similar configurations that vary in a single attribute such as the devices being targeted. For more information, see Cloning an Alert.

[top]


Alert Templates

There are a range of email templates that allow varying levels of customization to the alert configuration and alert email content.

Email (simple) - offers a single set of filters, basic alert bundling, and no email content customization



Email with custom content (simple) - offers a single set of filters, basic alert bundling, and email content customization



Email with custom content (advanced) - offers multiple filter sets, advanced bundling options, upstream device relationships and email content customization.

  • Multiple event filters: if any of the specified event filter sets is satisfied, an alert will be sent
  • Bundling: set a period within which to bundle alerts (events occurring outside of this period are not bundled) and a schedule for the sending of bundled alerts
  • Upstream Devices: suppress alerts for downstream devices when alerts exist for their upstream neighbors, see Upstream Device Configuration



Send syslog message - send an alert containing a syslog message

The syslog output can be in either the legacy Statseeker format, or an industry standard JSON format. The JSON format allows you to configure the message content to suit your requirements

The legacy format contains the following data:

  • Message number
  • Timestamp
  • Facility code
  • Application/Process
  • Loghost
  • Message

The JSON format includes all of the details from the Statseeker format and offers additional fields containing configuration data from the message source.

Sample basic syslog output for device down event:

010642 2019-08-28 15:29:51 NewYork-srv1 10.2.26.138 user.notice root: 10.124.1.1 Upstream-Neighbour-1 ping_state down,20671

Sample syslog JSON output for device down event with additional message elements selected:

010682 2019-08-28 15:32:51 NewYork-srv1 10.2.26.138 user.notice root:
{
"product": "Statseeker",
"version": "5.4.5.1908280800",
"source": "Upstream-Neighbour-1",
"event_type": "ping_state",
"state": "down",
"duration": 150,
"ipaddress": "10.124.1.1",
"sysLocation": "NewYork",
"sysName": "Upstream-Neighbour-1.example.com",
"sysDescr": "Cisco IOS Software, C2960 Software (C2960-LANBASEK9-M), Version 12.2(46)SE, RELEASE SOFTWARE (fc2) Copyright (c) 1986-2008 by Cisco Systems, Inc. Compiled Thu 21-Aug-08 15:59 by nachen",
"sysObjectID": "CISCO-PRODUCTS-MIB.catalyst2960G8TC",
"sysContact": "NewYork-support@example.com",
"snmp_version": 2,
"manual_name": null,
"latitude": 40.7306,
"longitude": -73.9352,
"hostname": null
}



Run a custom script - an event satisfying the filters will trigger the specified custom script.

Note: the alert process is handled by the statseeker system user account. The script can reside on the Statseeker server or another network location, but the statseeker account must have permission to both access and to execute the script.

[top]


Bundling Policies

Bundling refers to holding onto alerts for a specified period of time, and then sending all alerts generated over that period in a single alert email. This allows for a more efficient handling of events where several alerts are raised in a very short period of time, often in response to a single cause. Each alert configuration contains a bundling policy specific to that alert configuration, consequently, only alerts generated by that configuration are bundled according to that bundling policy.

Basic bundling options are available for the Email (simple) and Email with custom content (simple) templates. These options are:

  • No - no bundling occurs, and an alert is sent for each event
  • Yes - bundle the alerts for a specified duration
  • The duration of the bundling period

Once an event occurs and an alert is generated, Statseeker will hold onto the alert, and collect any other alert generated from the same alert configuration, until the specified bundling period is complete. At that time, a single alert email is sent detailing every alert collected over the bundling period.

Advanced bundling options are available for the Email with custom content (advanced) template. The bundling options available here are:

  • No - no bundling occurs, and an alert is sent for each event
  • Yes, with a duration - as per the bundling options offered with the simple templates
  • Yes, choose bundle times

When the bundle times option is selected the following fields are made available:

  • Bundle from - the time of day to begin bundling
  • Send email at - the time of day and days of the week, to stop bundling and send an alert email

In addition, the advanced template offers a Waiting field. This field is used to, optionally, specify an alternate action to be taken when device up/down events are received. The options here are:

  • No - do not wait, send alerts on down events as they are received
  • Yes, with a duration - wait for the specified duration prior to sending alerts on device down events. This option exposes a field to specify the duration.
  • Yes, wait over a specified period - this option exposes several additional fields
    • Behavior
      • Non-matching events sent as normal - events other than ping up/down are sent as normal during the specified waiting period
      • Only send up/down events relating to this period - ping up/down events outside of the specified period do not generate alerts
    • Start waiting - the start-time for the waiting policy
    • Finish waiting - the end-time for the waiting policy

[top]


Email Variables

Statseeker uses variables to enable customization of email subject and body content (requires Template = Email with custom content). The variable list is broken into 2 sections:

  • General - details from the event record
  • Data Type Specific - details relating to the entity (device, interface, etc.) which generated the event
Note:

  • When the selected Data Type is linked to another data type, fields from both are available as variables, For example if Data Type = Interface: the interface variable list includes a "Link to the parent device" option, allowing the configuration to include device variables as well.
  • When Alert Type = Threshold, the variable categories available in alerting are General and the Data Type the Threshold is configured against. If the Threshold specifies a Device Aggregation Format, then the available variables categories will instead be General and Device.

[top]

General Variables

The General variable list is common to all alert configurations using an email template. These items allow you to include variables referencing the event record which triggers the alert.

  • Event Time - the time the event occurred
  • Entity Name - the entity reporting the event, typically this is the Statseeker server but in some instances it can be a remote poller such as a Statseeker Observability Appliance
  • Device Name - name of the device creating the event record
  • Entity Type - the Data Type of the entity creating the event record
  • Text - event description
  • State - event state which triggered the event record creation
  • Down Duration - time since the State last changed (not available when State includes Down)
  • Threshold State – (Requires Alert Type = Threshold) the state of the breach (above, below, unknown)
  • Threshold Breach - (Requires Alert Type = Threshold) the breach value of the record

[top]

Inserting Variables

Inserting variables into the subject line



Inserting variables into the email body



Referencing variables in links

Variables can be added to link configurations allowing you to apply filters when linking through to Statseeker reports and dashboards. This process requires that you:

  • Run the dashboard or report and copy the URL
  • Edit the URL substituting report filter values or adding dashboard variable references as needed
Example: Adding a device filter to a report and dashboard
  • Report: locate and update device=foo to device={{event.general.device}}
  • Dashboard: add var-Device={{event.general.device}}, if var-Device is already in the URL, then update the value

Note: the syntax required when adding URL parameters is https://my.ip.or.domain?param1=foo&param2=bar...&paramN=foobar.

  • ? prefixes the first parameter, and additional parameters are prefixed with &

[top]

Variables and Bundled Alerts

When bundling is enabled, email content featuring variables is repeated for every instance of the alert being triggered within the bundling period. This includes:

  • Each table row which includes at least 1 variable
  • Each list item which includes at least 1 variable
  • Each paragraph of text which includes at least 1 variable
    • The Enter key will generate a new paragraph
    • Shift+Enter key will insert a line break without generating a new paragraph


[top]

Invalid Variables

If variables have been added to the email content\subject during alert configuration, and then the alert Data Type is changed such that the specified variables are not available to the updated Data Type, then variable is 'invalid' for the configured alert. Invalid variables will be returned as empty strings in the email.

[top]

Example Alert Configurations

Statseeker's Alerting allows for great variability in alerting configurations. The available configuration options vary with the selected Template, Alert Type, and Data Type. A complete walk-through of these options and their application is also available in video format.

A Ping-Down Alert using the Email (simple) template
  • Set Template = Email (simple)
  • Specify a Name for the alert configuration
  • Set:
    • Alert Type = Event
    • Data Type = Device
    • Field = Ping State - this action will display a list of ping states that can be alert on
  • Set States = down, optionally select additional ping states that will trigger this alert
  • Use the Entity/Group Filter to specify which devices on your network can trigger this alert
    • Group = All Groups - every monitored device
    • Group = <a selected group> - only devices in the selected group can trigger this alert
    • Device = <a selected device> - only the selected device can trigger this alert
  • Set a Time Filter, Time Filter Mode, and Time Zone as needed
    Note: When Time Filter = Custom an Advanced option is presented. This option assists with creating complex time filters, time filters with multiple exclusion periods, and offers the ability to test the time filter prior to use.
  • Optionally enable alert Bundling
  • Specify the alert recipient\s with the Email To setting
  • Optionally:
    • Check Event Filters - to confirm the correct configuration of event, entity\group, and time filters
    • Preview test Email - to preview the email content as it would be received by the alert recipient\s
    • Sent test Email - to confirm that the Statseeker server's email configuration allows for alert emails to be sent by the server and received by the recipient\s
  • Set Status = Enabled
  • Save the alert

Sending a Syslog message in response to a Threshold breach
  • Configure the Threshold (see Threshold Configuration for details)
  • Set Template = Syslog
  • Specify a Name for the alert configuration
  • Set the Time Zone as needed
  • Set Alert Type = Threshold, and select the Threshold from the list provided
  • Specify the threshold States which should trigger the alert
  • Specify Entity/Group Filters as needed
    • Group = All Groups - every monitored device
    • Group = <a selected group> - only devices in the selected group can trigger this alert
    • Device = <a selected device> - only the selected device can trigger this alert
    Note: filters specified here will operate in addition to any filters that exist within the Threshold configuration. Use Check Event Filters to confirm filter configurations.
  • Set a Time Filter, Time Filter Mode as needed
    Note: When Time Filter = Custom an Advanced option is presented. This option assists with creating complex time filters, time filters with multiple exclusion periods, and offers the ability to test the time filter prior to use.
  • Set Host = <target syslog aggregator IP address>
  • Specify the output syslog message Format (and if using JSON, specify any additional data to be included in the syslog output)
  • Set Status = Enabled
  • Save the alert


Editing Alerts

To edit an existing alert:

  • Select Administration Tool > Alerting / Event Management > Alerting
  • Select the alert from the list, populating the configuration panel
  • Modify the alert configuration as required
    • If you modify the alert filters, then use Check Event Filters to preview the impact of those changes
    • If you modify the alert email subject line or body content, then use Preview Test Email to preview the impact of those changes
  • Click Save Alert

[top]


Enabling/Disabling Alerts

To enable/disable alerts:

  • Select Administration Tool > Alerting / Event Management > Alerting
  • Click to select the alert/s and click Enable/Disable



To disable an alert from within the alert configuration:

  • Select Administration Tool > Alerting / Event Management > Alerting
  • Select the alert from the list, populating the configuration panel
  • Set Status to Disabled
  • Click Save Alert

The alert is now disabled, and no action will be taken based on this alert configuration until it is enabled again.

[top]


Cloning an Alert

To clone an existing alert configuration:

  • Select Administration Tool > Alerting / Event Management > Alerting
  • Select the configuration to be cloned
  • Click Clone

A new alert configuration will be created, but not saved, be sure to save the alert prior to leaving the configuration screen.

  • Modify the alert configuration as required
    • If you modify the alert filters, then use Check Event Filters to preview the impact of those changes
    • If you modify the alert email subject line or body content, then use Preview Test Email to preview the impact of those changes

[top]


Deleting Alerts

Deleting the alert configuration prevents the alert from being triggered and removes the alert configuration. To prevent an alert from being triggered but retain the alert configuration, Disable the alert rather than deleting it.

To delete an existing alert:

  • Select Administration Tool > Alerting / Event Management > Alerting
  • Select the alert/s from the list and click Delete
  • Click OK to confirm the action

The alert has now been removed from Statseeker.

[top]

Syslog, Trap, and Event Data Historical Records

While timeseries data is stored indefinitely, by default Statseeker stores syslog messages, SNMP Trap messages, and event (device, interface and threshold) records for a limited amount of time:

  • Syslog - 90 days
  • SNMP Traps - 90 days
  • Interface Events - 400 days
  • Threshold Events - 90 days
  • Device Events - 400 days

These values can be altered as needed and the records can be kept indefinitely if required (set storage time to 0). To update the default values:

  • Select Admin Tool > Network Discovery - Advanced Options > Advanced Options
  • Locate the settings in the History section and update as needed
  • Click Save


[top]