Index
- Overview
- Configuring Alerts
- Example Alert Configurations
- Editing Alerts
- Cloning an Alert
- Enabling/Disabling Alerts
- Deleting Alerts
- Syslog, Trap and Event Data Historical Records
Overview
Statseeker can be used as a comprehensive proactive, and reactive, alerting solution. Alerts can be triggered for:
- Threshold Events – thresholds can be configured against any timeseries metric monitored by Statseeker. By configuring thresholds Statseeker can send alerts when your network is experiencing unusual or unwanted activity such as excessive temperatures, memory\CPU load, or a device\link is nearing capacity. For more details, see Threshold Configuration.
- Device Events – device level events, such as ping state changes, on any devices monitored by Statseeker
- Interface Events – interface level events, such as ifOperStatus state changes, on any interfaces monitored by Statseeker
- Syslog Events – if your network contains devices that have been configured to output messages to a remote syslog server, then these devices can send those messages to Statseeker. Statseeker can review these logs and use the content to trigger alerts relating to those events.
- SNMP Trap Events – if your network contains devices that have been configured to output SNMP trap messages to Statseeker, then Statseeker can review these messages and trigger alerts based on their content.
Alerting is achieved through:
- Targeted Emails: a range of email templates and the ability to customize content makes it simple to provide the level of detail needed to respond to an issue efficiently. You will also need to configure Statseeker’s mail server to send email alerts, see Email Configuration for more information.
- Syslog Messages: Statseeker events can be used to trigger an alert which sends a syslog message to a collector\aggregator
- Custom Script: run a custom shell script in response to a specific alert trigger
Once an alert configuration has been enabled, it will be triggered by every subsequently recorded event satisfying the requirements for the alert. In some instances, such as during scheduled maintenance windows, you may wish to suppress some alerts. This can be achieved by disabling those alerts, see Disabling Alerts for more. When a more targeted approach is needed, Event Management rules can be defined to discard a specific type of event from a specified device (or group of devices). For more information, see Event Management.
Configuring Alerts
Statseeker’s Alerting allows for great variability in alerting configurations. The available configuration options vary with the selected Template, Alert Type, and Data Type. A complete walk-through of these options and their application is also available in video format.
To configure a new alert:
- Select Administration Tool > Alerting / Event Management > Alerting
This will display a list of currently configured alerts.
- Click Add
This will display the New Alert screen.
Field | Description |
Template | Alert template to use, for details on templates see Alert Templates
Note: the available templates can be edited from within the Admin Tool:
The allowed templates can be specified as a space\tab separated list with values of:
Example: |
Name | Name of the alert configuration |
Status |
|
Alert Type | What will trigger the alert:
Note: Statseeker allows for multiple ping pollers to be distributed throughout your network (see Observability Appliances for details). When multiple pollers are available, each device has a designated default poller, and can have any number of additional pollers monitoring the device. When:
|
Data Type | Requires Alert Type = Event; Syslog; Trap Select the data type the alert will be configured against. This is a filtered list only displaying those data types currently being monitored by your server. |
Threshold | Requires Alert Type = Threshold Select the threshold the alert will be configured against |
Field | Requires Data TypeSelect a state field associated with the selected Data Type |
States | Requires Field or Alert Type = ThresholdSelect the states which will trigger the alert |
Regex | Requires Alert Type = Syslog; TrapSpecify a RegEx to filter Syslog\Trap message records by content. Receipt of a message which satisfies the filter will result in the alert being triggered. |
Entity/Group Filter | Only events originating from the specified device or group can trigger alerts – not all entities are presented at once, begin typing to filter the list |
Time Filter | Matching events which occur within the specified time period will generate alerts. For more information on generating these, see Time Filters |
Time Filter Mode | Determines if the alert is triggered by events in effect at the beginning of the alerting period. E.g. You have an alert which references an ‘on transition’ threshold. If at the start of the alerting period (as specified by the Time Filter), the threshold is in breach.
|
Host | Requires Template = Syslog Specify the IP address of the target syslog collector\aggregator |
Format | Requires Template = Syslog Specify the output syslog message format
|
Time Zone | Used in conjunction with the time filter |
Enable Logging | Only enable logging when troubleshooting alerting issues |
Bundling | Requires Template = EmailBundling policy to be used, for details see Bundling Policies |
Waiting | Requires Template = Email with custom content (advanced)Specify an alternate bundling action to be taken when device ping state events are received, for details see Bundling Policies |
Upstream Devices | Requires Template = Email with custom content (advanced) Use your upstream device configurations to suppress ping/OperStatus down alerts from downstream devices when an alert exists for an upstream neighbor, see Upstream Device Configuration for details |
Email To | Requires Template = Email
|
Email Subject | Requires Template = Email Email subject line |
Mode | Requires Template = Email with custom content
Note:
|
Content | Requires Template = Email with custom content Specify the alert email content. This content can include text, links, images, tables, and variables. Note:
|
Recipient Roster File | Requires Template = Email with custom content (advanced) Specify a text file detailing an email recipient roster. Each line in the roster file should contain a start date-time followed by one or more email addresses. Recipient Roster Syntax
The format for the date-time is YYYY MM DD HH MM. The format for the roster file is: ** Ensure that there are no leading or trailing (blank) lines in the roster file ** Example: Once a date-time is reached, all future alerts generated from this configuration will be addressed to the specified email addresses. When the next date-time is reached, the recipients are replaced by those specified. |
Subject Date Format | Requires Template = Email with custom content (advanced) Select an alternate date-time format to be used for events times when a date-time variable is included in the email subject line |
Body Date Format | Requires Template = Email with custom content (advanced) Select an alternate date-time format to be used for events times when date-time variable is included in the email body |
Newlines | Requires Template = Email with custom content (advanced) Apply improved email body content layout for Windows end users, disable when email recipients are non-Windows users |
Check Event Filters | Runs the alert configuration against the Events database and returns recent events that would trigger the alert. |
Preview Test Email | Requires Template = Email Display a preview of the email content. This is useful when adding variables to your alert email subject line or body. |
Send Test Email | Test mail server and alert configuration by sending a test alert email |
Alert Templates
There are a range of email templates that allow varying levels of customization to the alert configuration and alert email content.
Email (simple) – offers a single set of filters, basic alert bundling, and no email content customization
Email with custom content (simple) – offers a single set of filters, basic alert bundling, and email content customization
Email with custom content (advanced) – offers multiple filter sets, advanced bundling options, upstream device relationships and email content customization.
- Multiple event filters: if any of the specified event filter sets is satisfied, an alert will be sent
- Bundling: set a period within which to bundle alerts (events occurring outside of this period are not bundled) and a schedule for the sending of bundled alerts
- Upstream Devices: suppress alerts for downstream devices when alerts exist for their upstream neighbors, see Upstream Device Configuration
Send syslog message – send an alert containing a syslog message
The syslog output can be in either the legacy Statseeker format, or an industry standard JSON format. The JSON format allows you to configure the message content to suit your requirements
The legacy format contains the following data:
- Message number
- Timestamp
- Facility code
- Application/Process
- Loghost
- Message
The JSON format includes all of the details from the Statseeker format and offers additional fields containing configuration data from the message source.
Sample basic syslog output for device down event:
010642 2019-08-28 15:29:51 NewYork-srv1 10.2.26.138 user.notice root: 10.124.1.1 Upstream-Neighbour-1 ping_state down,20671
Sample syslog JSON output for device down event with additional message elements selected:
010682 2019-08-28 15:32:51 NewYork-srv1 10.2.26.138 user.notice root:
{
“product”: “Statseeker”,
“version”: “5.4.5.1908280800”,
“source”: “Upstream-Neighbour-1”,
“event_type”: “ping_state”,
“state”: “down”,
“duration”: 150,
“ipaddress”: “10.124.1.1”,
“sysLocation”: “NewYork”,
“sysName”: “Upstream-Neighbour-1.example.com”,
“sysDescr”: “Cisco IOS Software, C2960 Software (C2960-LANBASEK9-M), Version 12.2(46)SE, RELEASE SOFTWARE (fc2) Copyright (c) 1986-2008 by Cisco Systems, Inc. Compiled Thu 21-Aug-08 15:59 by nachen”,
“sysObjectID”: “CISCO-PRODUCTS-MIB.catalyst2960G8TC”,
“sysContact”: “NewYork-support@example.com”,
“snmp_version”: 2,
“manual_name”: null,
“latitude”: 40.7306,
“longitude”: -73.9352,
“hostname”: null
}
Run a custom script – an event satisfying the filters will trigger the specified custom script.
Bundling Policies
Bundling refers to holding onto alerts for a specified period of time, and then sending all alerts generated over that period in a single alert email. This allows for a more efficient handling of events where several alerts are raised in a very short period of time, often in response to a single cause. Each alert configuration contains a bundling policy specific to that alert configuration, consequently, only alerts generated by that configuration are bundled according to that bundling policy.
Basic bundling options are available for the Email (simple) and Email with custom content (simple) templates. These options are:
- No – no bundling occurs, and an alert is sent for each event
- Yes – bundle the alerts for a specified duration
- The duration of the bundling period
Once an event occurs and an alert is generated, Statseeker will hold onto the alert, and collect any other alert generated from the same alert configuration, until the specified bundling period is complete. At that time, a single alert email is sent detailing every alert collected over the bundling period.
Advanced bundling options are available for the Email with custom content (advanced) template. The bundling options available here are:
- No – no bundling occurs, and an alert is sent for each event
- Yes, with a duration – as per the bundling options offered with the simple templates
- Yes, choose bundle times
When the bundle times option is selected the following fields are made available:
- Bundle from – the time of day to begin bundling
- Send email at – the time of day and days of the week, to stop bundling and send an alert email
In addition, the advanced template offers a Waiting field. This field is used to, optionally, specify an alternate action to be taken when device up/down events are received. The options here are:
- No – do not wait, send alerts on down events as they are received
- Yes, with a duration – wait for the specified duration prior to sending alerts on device down events. This option exposes a field to specify the duration.
- Yes, wait over a specified period – this option exposes several additional fields
- Behavior
- Non-matching events sent as normal – events other than ping up/down are sent as normal during the specified waiting period
- Only send up/down events relating to this period – ping up/down events outside of the specified period do not generate alerts
- Start waiting – the start-time for the waiting policy
- Finish waiting – the end-time for the waiting policy
- Behavior
Email Variables
Statseeker uses variables to enable customization of email subject and body content (requires Template = Email with custom content). The variable list is broken into 2 sections:
- General – details from the event record
- Data Type Specific – details relating to the entity (device, interface, etc.) which generated the event
- When the selected Data Type is linked to another data type, fields from both are available as variables, For example if Data Type = Interface: the interface variable list includes a “Link to the parent device” option, allowing the configuration to include device variables as well.
- When Alert Type = Threshold, the variable categories available in alerting are General and the Data Type the Threshold is configured against. If the Threshold specifies a Device Aggregation Format, then the available variables categories will instead be General and Device.
General Variables
The General variable list is common to all alert configurations using an email template. These items allow you to include variables referencing the event record which triggers the alert.
- Event Time – the time the event occurred
- Entity Name – the entity reporting the event, typically this is the Statseeker server but in some instances it can be a remote poller such as a Statseeker Observability Appliance
- Device Name – name of the device creating the event record
- Entity Type – the Data Type of the entity creating the event record
- Text – event description
- State – event state which triggered the event record creation
- Down Duration – time since the State last changed (not available when State includes Down)
- Threshold State – (Requires Alert Type = Threshold) the state of the breach (above, below, unknown)
- Threshold Breach – (Requires Alert Type = Threshold) the breach value of the record
Inserting Variables
Inserting variables into the subject line
Inserting variables into the email body
Referencing variables in links
Variables can be added to link configurations allowing you to apply filters when linking through to Statseeker reports and dashboards. This process requires that you:
- Run the dashboard or report and copy the URL
- Edit the URL substituting report filter values or adding dashboard variable references as needed
Example: Adding a device filter to a report and dashboard
- Report: locate and update device=foo to device={{event.general.device}}
- Dashboard: add var-Device={{event.general.device}}, if var-Device is already in the URL, then update the value
- ? prefixes the first parameter, and additional parameters are prefixed with &
Variables and Bundled Alerts
When bundling is enabled, email content featuring variables is repeated for every instance of the alert being triggered within the bundling period. This includes:
- Each table row which includes at least 1 variable
- Each list item which includes at least 1 variable
- Each paragraph of text which includes at least 1 variable
- The Enter key will generate a new paragraph
- Shift+Enter key will insert a line break without generating a new paragraph
Invalid Variables
If variables have been added to the email content\subject during alert configuration, and then the alert Data Type is changed such that the specified variables are not available to the updated Data Type, then variable is ‘invalid’ for the configured alert. Invalid variables will be returned as empty strings in the email.
Example Alert Configurations
Statseeker’s Alerting allows for great variability in alerting configurations. The available configuration options vary with the selected Template, Alert Type, and Data Type. A complete walk-through of these options and their application is also available in video format.
A Ping-Down Alert using the Email (simple) template
- Set Template = Email (simple)
- Specify a Name for the alert configuration
- Set:
- Alert Type = Event
- Data Type = Device
- Field = Ping State – this action will display a list of ping states that can be alert on
- Set States = down, optionally select additional ping states that will trigger this alert
- Use the Entity/Group Filter to specify which devices on your network can trigger this alert
- Group = All Groups – every monitored device
- Group = <a selected group> – only devices in the selected group can trigger this alert
- Device = <a selected device> – only the selected device can trigger this alert
- Set a Time Filter, Time Filter Mode, and Time Zone as needed
Note: When Time Filter = Custom an Advanced option is presented. This option assists with creating complex time filters, time filters with multiple exclusion periods, and offers the ability to test the time filter prior to use.
- Optionally enable alert Bundling
- Specify the alert recipient\s with the Email To setting
- Optionally:
- Check Event Filters – to confirm the correct configuration of event, entity\group, and time filters
- Preview test Email – to preview the email content as it would be received by the alert recipient\s
- Sent test Email – to confirm that the Statseeker server’s email configuration allows for alert emails to be sent by the server and received by the recipient\s
- Set Status = Enabled
- Save the alert
Sending a Syslog message in response to a Threshold breach
- Configure the Threshold (see Threshold Configuration for details)
- Set Template = Syslog
- Specify a Name for the alert configuration
- Set the Time Zone as needed
- Set Alert Type = Threshold, and select the Threshold from the list provided
- Specify the threshold States which should trigger the alert
- Specify Entity/Group Filters as needed
- Group = All Groups – every monitored device
- Group = <a selected group> – only devices in the selected group can trigger this alert
- Device = <a selected device> – only the selected device can trigger this alert
Note: filters specified here will operate in addition to any filters that exist within the Threshold configuration. Use Check Event Filters to confirm filter configurations. - Set a Time Filter, Time Filter Mode as needed
Note: When Time Filter = Custom an Advanced option is presented. This option assists with creating complex time filters, time filters with multiple exclusion periods, and offers the ability to test the time filter prior to use.
- Set Host = <target syslog aggregator IP address>
- Specify the output syslog message Format (and if using JSON, specify any additional data to be included in the syslog output)
- Set Status = Enabled
- Save the alert
Editing Alerts
To edit an existing alert:
- Select Administration Tool > Alerting / Event Management > Alerting
- Select the alert from the list, populating the configuration panel
- Modify the alert configuration as required
- If you modify the alert filters, then use Check Event Filters to preview the impact of those changes
- If you modify the alert email subject line or body content, then use Preview Test Email to preview the impact of those changes
- Click Save Alert
Enabling/Disabling Alerts
To enable/disable alerts:
- Select Administration Tool > Alerting / Event Management > Alerting
- Click to select the alert/s and click Enable/Disable
To disable an alert from within the alert configuration:
- Select Administration Tool > Alerting / Event Management > Alerting
- Select the alert from the list, populating the configuration panel
- Set Status to Disabled
- Click Save Alert
The alert is now disabled, and no action will be taken based on this alert configuration until it is enabled again.
Cloning an Alert
To clone an existing alert configuration:
- Select Administration Tool > Alerting / Event Management > Alerting
- Select the configuration to be cloned
- Click Clone
A new alert configuration will be created, but not saved, be sure to save the alert prior to leaving the configuration screen.
- Modify the alert configuration as required
- If you modify the alert filters, then use Check Event Filters to preview the impact of those changes
- If you modify the alert email subject line or body content, then use Preview Test Email to preview the impact of those changes
Deleting Alerts
Deleting the alert configuration prevents the alert from being triggered and removes the alert configuration. To prevent an alert from being triggered but retain the alert configuration, Disable the alert rather than deleting it.
To delete an existing alert:
- Select Administration Tool > Alerting / Event Management > Alerting
- Select the alert/s from the list and click Delete
- Click OK to confirm the action
The alert has now been removed from Statseeker.
Syslog, Trap, and Event Data Historical Records
While timeseries data is stored indefinitely, by default Statseeker stores syslog messages, SNMP Trap messages, and event (device, interface and threshold) records for a limited amount of time:
- Syslog – 90 days
- SNMP Traps – 90 days
- Interface Events – 400 days
- Threshold Events – 90 days
- Device Events – 400 days
These values can be altered as needed and the records can be kept indefinitely if required (set storage time to 0). To update the default values:
- Select Admin Tool > Network Discovery – Advanced Options > Advanced Options
- Locate the settings in the History section and update as needed
- Click Save