CH logo® Knowledge Base
Contents Search
   

 

Alert Notifications

The configuration of notifications is covered in the following topics:

Notes:
- For general information on policy alerting, see Policy Alert Overview.
- For information on settings for alert policies, see Alert Policies.
- For information on active or historical alerts, see Alert Dashboards.
- For information on mitigation for alerts, see Alert Mitigation.

 

 
 top

About Notification Channels

The Channels page enables the creation and management of notification channels. Each channel represents a notification mode (e.g. email) and one or more notification targets (e.g. a set of email addresses). Notification channels are assigned in an alert policy (see Threshold Notify Settings) to determine who will be notified when there’s a change in the alert state, e.g. an alarm is triggered by one of the alert’s thresholds.

 

 
 top

Channels Page UI

The Channels page is made up of the following UI elements:

  • Filter field: Filters the Notification Channel List to show only rows containing the entered text in one of the following fields: ID, Name, Channel Type, Destination.
  • Add Notification Channel: A button that opens the Add Notification Channel dialog (see Add or Edit Channel).
  • Notification Channel List: A list of the notification channels existing in your organization; see Notification Channel List.

 

 
 top

Notification Channel List

The Notification Channel List is a table that lists all of the alert notification channels that have been created by users in your organization. The table includes the following columns:

  • ID: System-assigned ID for the notification channel.
  • Name: User-assigned name for the channel.
  • Channel Type: The type of notification channel, which may be one of the following:
    - Email
    - JSON POST Webhook
    - OpsGenie
    - PagerDuty
    - ServiceNow
    - Slack
    - System Log
  • Destination: The target(s) to which the notification will be directed. For email, this is one or more email addresses.
  • Remove (trash icon): Deletes the notification channel.
    Note: The platform is deleted immediately (no confirming dialog).

Note: To edit a channel (see Add or Edit Channel), click on its row in the list, which opens the Edit Notification Channel dialog.

 

 
 top

Add or Edit Channel

Notification channels are added or edited using one of the following dialogs:

  • Add Notification Channel: Used to create a new alert notification channel. Access via the Add Notification Channel button at the upper right of the Channels page.
  • Edit Notification Channel: Used to modify an existing notification channel. Access by clicking in any individual row of the Notification Channel List.

The configuration of notifications in these dialogs depends on the notification type and involves the following steps, which are covered in the topics below:

  • Set the Common Notification Settings.
  • Set any additional settings specific to the notification type.
  • Follow any additional steps external to Kentik Detect that are required to complete setup of the notification channel.

 

 
 top

Notification Dialogs UI

The Add Notification Channel and Edit Notification Channel dialogs share the same layout and the following common UI elements:

  • Close button: Click the X in the upper right corner to close the dialog. All elements will be restored to their values at the time the dialog was opened.
  • Remove button (Edit Mitigation Platform dialog only): Remove the channel from your organization’s collection of channels.
  • Cancel button: Cancel the add channel or edit channel operation and exit the dialog. All elements will be restored to their values at the time the dialog was opened.
  • Add Notification Channel button (Add Notification Channel dialog only): Save settings for the new channel and exit the dialog.
  • Save button (Edit Notification Channel dialog only): Save changes to channel settings and exit the dialog.

 

 
 top

Notification Channel Settings

The settings of the Add Notification Channel and Edit Notification Channel dialogs depend on the type of channel you are trying to configure. These settings are covered in the following topics:

 

 
 top  |  section

Common Notification Settings

The following settings are common to all notification types:

  • Name: A user-assigned name for the notification channel.
  • Notification Type: A drop-down menu for choosing the type of alert notification channel. The remaining fields of the dialog vary depending on the chosen type, which may be one of the following:
    - Email
    - JSON POST Webhook
    - System Log (syslog)
    - Slack Channel
    - PagerDuty

Once the notification type has been selected the dialog will display the remaining settings for that type, which are covered in the following topics.

 

 
 top  |  section

Email Notification Settings

The following settings are used only for email alert notifications:

  • Email(s): A comma-delimited list of email addresses to which notifications will be sent.
  • Status Digest Interval: Sets the interval at which an email will be sent listing items (alarms or mitigations) currently in an active state. Click the field to choose from a drop[-down list.

The following alarm and mitigation states (see Alert States) are considered active:

  • Alarm active state: ALARM
  • Mitigation active states: MITIGATING, END_WAIT_CONF, END_TIMED_CONF, END_GRACE, MITIGATING_MANUAL, ROGUE_MITIGATION_FAIL.

The image below shows a representative alert notification email from Kentik Detect.

 

 
 top  |  section

JSON Notification Settings

The following setting is used only for JSON alert notifications via a webhook URL to which JSON can be posted:

  • Webhook URL: The URL to which Kentik Detect should post JSON notifications for this channel.

When the notification type is set to JSON POST Webhook the Add Notification Channel or Edit Notification Channel dialog will also display a Webhook Security Information field stating the user-agent HTTP Header and the IP/CIDR from which notifications will be sent.

Note: The specified Webhook URL:
- may use either HTTP or HTTPS;
- may include a port if you prefer not to use the default HTTP TCP/80;
- must be reachable (i.e. won’t be dropped by a firewall between Kentik Detect and your system) from the IP subnet shown in the Webhook Security Information field.

 

Using JSON Notifications

JSON notifications enable you to integrate Kentik Detect with third-party monitoring systems so that Kentik anomaly detection alarms can trigger external actions, which may include configurations, DDoS attack mitigations, or other remedies. The JSON is posted to the specified Webhook URL, where it can be parsed and processed for any desired purpose. For an example of the JSON that will be posted, see Sample Alert JSON.

 

Secure Receipt of JSON

You can secure your receipt of JSON alert notifications from Kentik Detect using any of the following complementary methods:

  • Filter by IP: Filter inbound POST requests (using iptables on the web-service server or any firewall in the path, or in the code of the web service itself) to only those from IPs in the netblock 209.50.158.0/23.
  • Filter by HTTP header: Filter inbound POST requests to only those with the following header: “User-Agent:KentikAlerting”
  • Use a query argument: If you use HTTPS for the Webhook, include a query argument known only to your responding web-service.

 

Testing JSON Notifications

Issues that often arise when trying to test POST requests include having a web server that’s available to accept the requests before your web service is in production and having a service that is available on a public URL. Kentik suggests trying the methods below to address these issues.

  • RequestBin: Using the free web-service RequestBin (https://requestb.in/), you’ll be able to collect HTTP requests in bins that you can examine to see what is being posted to them.
  • ngrok: If you have a development web server on your local host that accepts HTTP-POST, try using ngrok (https://ngrok.com/) to get a unique public URL that is directly connected to your development machine.

 

Sample Alert JSON

The following example (with placeholders highlighted) illustrates the JSON that would typically be sent in response to a change in the state of an alert.

{
  "EventType":"ALARM_STATE_CHANGE",
  "CompanyID":####,
  "MitigationID":0,
  "AlarmID":#####,
  "AlarmState":"ALARM",
  "PolicyID":####,
  "ThresholdID":####,
  "ActivateSeverity":"major",
  "AlarmStart":"2017-05-08T18:17:18Z",
  "AlarmEnd":"0001-01-01T00:00:00Z",
  "LastActivate":"2017-05-08T18:17:18Z",
  "AlertPolicyName":"TEST_ALWAYS_FIRE",
  "AlarmStateOld":"",
  "AlertKey":[
    {
      "DimensionName":"IP_dst",
      "DimensionValue":"###.###.###.#"
    }
  ],
  "AlertValue":{
    "Unit":"Packets/s",
    "Value":21760
  },
  "AlertBaseline":{
    "Unit":"Packets/s",
    "Value":1000
  }
  "AlertBaselineSource": "ACT_BASELINE_MISSING_DEFAULT_INSTEAD_OF_LOWEST"
}

 

Alert JSON Description

The following table explains the elements of the notification JSON:

JSON element Description Type and values
EventType The type of message, either alarm or mitigation notification. String: ALARM_STATE_CHANGE, MITIGATION_STATE_CHANGE
CompanyID Your company ID Number
MitigationID ID of mitigation, if any. If no mitigation, then 0. Number
AlarmID ID of alarm. Number
AlarmState Current state of alarm. String. See Alert States.
PolicyID ID of the alert policy generating this notification. Number
ThresholdID ID of the specific threshold (within a policy) that was triggered to generate the alarm. Number
ActivateSeverity Severity level of the threshold generating the alarm (see General Threshold Settings). String: Minor, Minor2, Major, Major2, Critical
AlarmStart A date-time string giving the time that the alarm started. String
AlarmEnd A date-time string giving the time that the alarm ended. If not yet ended, time will be given as 0001-01-01T00:00:00Z. String
LastActivate A date-time string giving the last time that this alarm was active. String
AlertPolicyName The name of the alert policy generating this notification. String
AlarmStateOld The prior state of this alarm. See Alert States.
AlertKey An array of the dimensions that make up the key used to evaluate traffic for this alert policy (see About Keys). Array
AlertKey » DimensionName The system name of an individual dimension that is part of this policy’s key. String. See Main Table Schema.
AlertKey » DimensionValue The value of an individual dimension that is part of this policy’s key. String or number depending on dimension.
AlertValue An array of current traffic metrics (primary and secondary) by which ingested flow data will be evaluated to determine top-X. Array
AlertValue » Unit The UI name of an individual current-traffic metric. String. See General Metrics and Host Traffic Metrics.
AlertValue » Value The value of an individual current-traffic metric. Number
AlertBaseline An array of baseline traffic metrics (primary and secondary) by which ingested flow data will be evaluated to determine top-X. Array
AlertBaseline » Unit The UI name of an individual baseline metric. String. See General Metrics and Host Traffic Metrics.
AlertBaseline » Value The value of an individual baseline metric. Number
AlertBaselineSource The source of the baseline information in the AlertBaseline array. String. See Baseline Source for Notifications.

 

Baseline Source for Notifications

The following table shows the source of the information returned in the AlertBaselineSource element of the notification JSON:

Baseline Source Name Description Baseline Mode
NO_USE_BASELINE Baseline not used on this threshold at all. Current to History
SKIPED_BASELINE_CALCULATION Baseline fallback set to “Do Not Alarm.” Current to History
TRIGGER_USED_NO_BASELINE Baseline fallback set to “Alarm.” Current to History
CALCULATED_USED_FOR_BASELINE Baseline fallback used “Auto Calc” value Current to History
DEFAULT_USED_FOR_BASELINE Baseline fallback used Static value Current to History
LOWEST_USED_FOR_BASELINE Baseline fallback used the “Lowest Top-X” setting Current to History
NOT_FOUND_EXISTS_NO_BASELINE Baseline not found and no fallback option set. Current to History
ACT_CURRENT_MISSING_TRIGGER Key was in the Historical Top-X but not in the Current Top-X. History to Current
ACT_CURRENT_USED_FOUND Key was found in both Historical and Current Top-X, threshold exceeded. History to Current
ACT_CURRENT_MISSING_DEFAULT Key was in the Historical Top-X but not in the Current Top-X. History to Current
ACT_CURRENT_MISSING_LOWEST Key was in the Historical Top-X but not in the Current Top-X, lowest Top-X fallback History to Current

 

 
 top  |  section

OpsGenie Notification Settings

The following setting is used only for OpsGenie alert notifications:

  • OpsGenie GenieKey: A unique service identifier used by the OpsGenie WebAPI.

Before establishing an OpsGenie notification channel in Kentik Detect you will need to set up a corresponding “integration” in OpsGenie, which is described in the OpsGenie documentation topic Using API Integration. Assuming that you already have an OpsGenie account, you can add an integration for an existing team or create a new team of individual OpsGenie users that you specify (see the OpsGenie topic Create a Team).

Once you have a team, adding an integration involves the following steps:

  1. In OpsGenie, go to the Integrations tab of your Team Dashboard.
  2. Click the Add Integration button. The page will fill with a set of tiles, each representing an integration.
  3. In the API tile, click the Add button. An Integration page will open (by default, the title will be your team name with “_API” appended).
  4. Copy the API Key field. You’ll need to enter this key into the OpsGenie GenieKey field when you configure an OpsGenie notification channel in Kentik Detect.
  5. Click the Save Integration button to save the Integration.

When you configure an OpsGenie channel in Kentik Detect and assign it to an Alert Policy (see Threshold Activation), notifications from thresholds that trigger alarms will appear in the OpsGenie Alert List. Click on an item in the list to see an Alert Detail (shown below), which contains effectively the same information that’s included in a JSON alert notification (see Sample Alert JSON).

 

 
 top  |  section

PagerDuty Notification Settings

The following setting is used only for PagerDuty alert notifications:

  • PagerDuty Integration Key: A unique service identifier used by the PagerDuty Events API to trigger, acknowledge, and resolve incidents.

Before establishing a PagerDuty notification in Kentik Detect you will want to set up a corresponding service in PagerDuty (see PagerDuty’s Configuring Services and Integrations support page):

  1. Configure at least one Service to hook notifications to. If desired, you can create multiple Services that go to different people depending on the type of alert.
  2. Create a New Integration using the “Use our API Directly” option for the Integration Type.
  3. On PagerDuty’s Service page, associate the Integration with the Service, choosing Events APIv2 in the drop-down list.
  4. Note the Integration Key from the Integrations tab of the Service Details screen for the desired service. You’ll use the key in the Create/Edit Notification Channel dialog in Kentik Detect.

Note: The data included in a PagerDuty alert notification from Kentik Detect is effectively the same as what’s included in a JSON alert notification (see Sample Alert JSON).

 

 
 top  |  section

ServiceNow Notification Settings

The following settings are used only for ServiceNow alert notifications:

  • ServiceNow Instance Name: A unique identifier used by ServiceNow.
  • ServiceNow Username: The username associated with the ServiceNow Instance.
  • ServiceNow Password: The password associated with the ServiceNow Instance.

Before establishing a ServiceNow notification channel in Kentik Detect you will need to register for ServiceNow, log into your account, and request a ServiceNow “instance” (see the ServiceNow training topic Personal Developer Instances). Getting the information you need to configure ServiceNow in Kentik Detect involves the following steps:

  1. In ServiceNow, choose Instance from the Manage menu.
  2. Click the Request Instance button and specify a ServiceNow version for the instance.
  3. The page will update to display the name, username, and password for the new instance. You’ll need to enter this information into the fields listed above when you configure a ServiceNow notification channel in Kentik Detect.

Once you configure an ServiceNow channel in Kentik Detect and assign it to an Alert Policy (see Threshold Activation), notifications from thresholds that trigger alarms will be listed on the ServiceNow Incidents page. Click on an item in the list to see the details page for an incident (shown below), which effectively contains the same information that’s included in a JSON alert notification (see Sample Alert JSON).

 

 
 top  |  section

Slack Notification Settings

Slack channel alert notifications use only the Common Notification Settings in the Add Notification Channel or Edit Notification Channel dialog. In addition to settings in the dialog, however, setup of Slack alert notifications also involves configuration in Slack itself:

  1. Before establishing a Slack notification in Kentik Detect, first create the channel in Slack on which you wish to receive notifications from Kentik Detect.
    Note: Confirm the desired Public/Private setting of the new Slack channel before continuing to step 2.
  2. In the Kentik Detect dialog (either Add or Edit), select Slack as the notification type, enter a name for the channel, and then click the Save button. You will be taken to Slack to complete setup of the channel.
  3. In Slack, you’ll be asked to allow the Kentik Alerts application to access your “team” (your company or organization).
  4. Still in Slack, you’ll be asked to select the Slack channel that the Kentik Alerts application can post to. Choose the Slack channel that you created in step 1. The channel that you named in step 2 is now associated with that Slack channel.

 

Slack Notification Troubleshooting

If you created a Slack notification channel from within the Kentik portal but are not receiving notifications, troubleshoot as follows:

  1. Contact your organization’s Slack administrator and have them verify:
    - your allowed applications;
    - the permissions that enable you to add integrations and applications.
  2. In Kentik Detect, in the Notification Channel List on the Channels page (Alerting » Notification Channels), find the channel that you created (in step 2 of Slack Notification Settings) to receive Slack notifications from Kentik. Use the Remove button (trash icon) at the right of the row to remove the channel.
  3. Redo steps 2 through 4 in the Slack Notification Settings topic.
  4. If the issue persists, repeat step 2 above until you have removed all existing notification channels that use Slack. Then recreate each of the deleted channels (see steps 2 through 4 in Slack Notification Settings).
Note: Removing existing Slack-based notification channels will affect all alert policies in your organization that use Slack notifications. After you remove and recreate the channels you will have to add them back to every policy threshold in which they were previously assigned (see Threshold Activation).

Problems with Slack notification can be caused by changing the Public/Private setting of a Slack channel in your organization’s Slack system after that channel has been assigned to a Kentik notification channel (e.g. integrating a channel that’s Public, then switching that channel to Private). If you need to change the setting, first remove the notification channel from Kentik, then modify the Slack channel, and then re-create the notification channel in Kentik.

 

 
 top  |  section

Syslog Notification Settings

The following settings are used only for System Log (syslog) alert notifications:

  • Destination IP: IP address to which Kentik Detect notification messages will be posted.
  • Port: Port on which syslog will listen for notifications.
  • Transport Type: Protocol, either UDP or TCP
  • Hostname: Name for the syslog to which messages will be posted.
  • Severity: The severity level of the notification messages sent via this channel. Options include emergency, alert, critical, etc. as defined in RFC 5424.
  • Facility: The facility code of the notification messages sent via this channel. Options include kernel, user, system, etc. as defined in RFC 5424.

When the notification type is set to System Log the Add Notification Channel or Edit Notification Channel dialog will also display a Syslog Security Information field stating the IP/CIDR from which notifications will be sent.

Note: The data included in a Syslog alert notification from Kentik Detect is effectively the same as what’s included in a JSON alert notification (see Sample Alert JSON).

 

In this article: