Alert Notifications

The configuration of notifications is covered in the following topics:

• About Notification Channels
• Channels Page UI
• Notification Channel List
• Add or Edit Channel
• Notification Dialogs UI
• Notification Channel Settings

Notes:
- For general information on policy alerting, see Policy Alert Overview.
- For information on settings for alert policies, see Alert Policies.
- For information on active or historical alerts, see Alert Dashboards.
- For information on mitigation for alerts, see Mitigation.

The Channels page enables the creation and management of notification channels. Each channel represents a notification mode (e.g. email) and one or more notification targets (e.g. a set of email addresses). Notification channels are assigned in an alert policy (see Threshold Notify Settings) to determine who will be notified when there’s a change in the alert state, e.g. an alarm is triggered by one of the alert’s thresholds.

The Channels page is made up of the following UI elements:

• Filter field: Filters the Notification Channel List to show only rows containing the entered text in one of the following fields: ID, Name, Channel Type, Destination.
• Add Notification Channel: A button that opens the Add Notification Channel dialog (see Add or Edit Channel).
• Notification Channel List: A list of the notification channels existing in your organization; see Notification Channel List.

The Notification Channel List is a table that lists all of the alert notification channels that have been created by users in your organization. The table includes the following columns:

• ID: System-assigned ID for the notification channel.
• Name: User-assigned name for the channel.
• Channel Type: The type of notification channel, which may be one of the following:
  - Email
  - JSON POST Webhook
  - OpsGenie
  - PagerDuty
  - ServiceNow
  - Slack
  - System Log
• Destination: The target(s) to which the notification will be directed. For email, this is one or more email addresses.
• Remove (trash icon): Deletes the notification channel.
  Note: The platform is deleted immediately (no confirming dialog).

Note: To edit a channel (see Add or Edit Channel), click on its row in the list, which opens the Edit Notification Channel dialog.

Notification channels are added or edited using one of the following dialogs:

• Add Notification Channel: Used to create a new alert notification channel. Access via the Add Notification Channel button at the upper right of the Channels page.
• Edit Notification Channel: Used to modify an existing notification channel. Access by clicking in any individual row of the Notification Channel List.

The configuration of notifications in these dialogs depends on the notification type and involves the following steps, which are covered in the topics below:

• Set the Common Notification Settings.
• Set any additional settings specific to the notification type.
• Follow any additional steps external to Kentik Detect that are required to complete setup of the notification channel.

The Add Notification Channel and Edit Notification Channel dialogs share the same layout and the following common UI elements:

• Close button: Click the X in the upper right corner to close the dialog. All elements will be restored to their values at the time the dialog was opened.
• Remove button (Edit Mitigation Platform dialog only): Remove the channel from your organization’s collection of channels.
• Cancel button: Cancel the add channel or edit channel operation and exit the dialog. All elements will be restored to their values at the time the dialog was opened.
• Add Notification Channel button (Add Notification Channel dialog only): Save settings for the new channel and exit the dialog.
• Save button (Edit Notification Channel dialog only): Save changes to channel settings and exit the dialog.

The settings of the Add Notification Channel and Edit Notification Channel dialogs depend on the type of channel you are trying to configure. These settings are covered in the following topics:

• Common Notification Settings
• Email Notification Settings
• JSON Notification Settings
• OpsGenie Notification Settings
• PagerDuty Notification Settings
• Slack Notification Settings
• ServiceNow Notification Settings
• Syslog Notification Settings

The following settings are common to all notification types:

• Name: A user-assigned name for the notification channel.
• Notification Type: A drop-down menu for choosing the type of alert notification channel. The remaining fields of the dialog vary depending on the chosen type, which may be one of the following:
  - Email
  - JSON POST Webhook
  - System Log (syslog)
  - Slack Channel
  - PagerDuty

Once the notification type has been selected the dialog will display the remaining settings for that type, which are covered in the following topics.

The following settings are used only for email alert notifications:

• Email(s): A comma-delimited list of email addresses to which notifications will be sent.
• Status Digest Interval: Sets the interval at which an email will be sent listing items (alarms or mitigations) currently in an active state. Click the field to choose from a drop-down list.

The following alarm and mitigation states (see Alert States) are considered active:

• Alarm active state: ALARM
• Mitigation active states: MITIGATING, END_WAIT_CONF, END_TIMED_CONF, END_GRACE, MITIGATING_MANUAL, ROGUE_MITIGATION_FAIL.

The image below shows a representative alert notification email from Kentik Detect.

Note: In each row of a notification email’s list of active alerts there are links that enable you to view the traffic that triggered the alarm, either in Data Explorer or in the designated “policy dashboard” (see General Policy Settings).

The following setting is used only for JSON alert notifications via a webhook URL to which JSON can be posted:

• Webhook URL: The URL to which Kentik Detect should post JSON notifications for this channel.

When the notification type is set to JSON POST Webhook the Add Notification Channel or Edit Notification Channel dialog will also display a Webhook Security Information field stating the user-agent HTTP Header and the IP/CIDR from which notifications will be sent.

Note: The specified Webhook URL:
- may use either HTTP or HTTPS;
- may include a port if you prefer not to use the default HTTP TCP/80;
- must be reachable (i.e. won’t be dropped by a firewall between Kentik Detect and your system) from the IP subnet shown in the Webhook Security Information field.

JSON notifications enable you to integrate Kentik Detect with third-party monitoring systems so that Kentik anomaly detection alarms can trigger external actions, which may include configurations, DDoS attack mitigations, or other remedies. The JSON is posted to the specified Webhook URL, where it can be parsed and processed for any desired purpose. For an example of the JSON that will be posted, see Sample Alert JSON.

You can secure your receipt of JSON alert notifications from Kentik Detect using any of the following complementary methods:

• Filter by IP: Filter inbound POST requests (using iptables on the web-service server or any firewall in the path, or in the code of the web service itself) to only those from IPs in the netblock 209.50.158.0/23.
• Filter by HTTP header: Filter inbound POST requests to only those with the following header: “User-Agent:KentikAlerting”
• Use a query argument: If you use HTTPS for the Webhook, include a query argument known only to your responding web-service.

Issues that often arise when trying to test POST requests include having a web server that’s available to accept the requests before your web service is in production and having a service that is available on a public URL. Kentik suggests trying the methods below to address these issues.

• RequestBin: Using the free web-service RequestBin (https://requestbin.com), you’ll be able to collect HTTP requests in bins that you can examine to see what is being posted to them.
• ngrok: If you have a development web server on your local host that accepts HTTP-POST, try using ngrok (https://ngrok.com/) to get a unique public URL that is directly connected to your development machine.

The following example (with placeholders highlighted) illustrates the JSON that would typically be sent in response to a change in the state of an alert.

{
  "EventType":"ALARM_STATE_CHANGE",
  "CompanyID":####,
  "MitigationID":0,
  "AlarmID":#####,
  "AlarmState":"ALARM",
  "PolicyID":####,
  "ThresholdID":####,
  "ActivateSeverity":"major",
  "AlarmStart":"2017-05-08T18:17:18Z",
  "AlarmEnd":"0001-01-01T00:00:00Z",
  "LastActivate":"2017-05-08T18:17:18Z",
  "AlertPolicyName":"TEST_ALWAYS_FIRE",
  "AlarmStateOld":"",
  "AlertKey":[
    {
      "DimensionName":"IP_dst",
      "DimensionValue":"###.###.###.#"
    }
  ],
  "AlertValue":{
    "Unit":"Packets/s",
    "Value":21760
  },
  "AlertBaseline":{
    "Unit":"Packets/s",
    "Value":1000
  }
  "AlertBaselineSource": "ACT_BASELINE_MISSING_DEFAULT_INSTEAD_OF_LOWEST"
}

The following table explains the elements of the notification JSON:

The following table shows the source of the information returned in the AlertBaselineSource element of the notification JSON:

The following setting is used only for OpsGenie alert notifications:

• OpsGenie GenieKey: A unique service identifier used by the OpsGenie WebAPI.

Before establishing an OpsGenie notification channel in Kentik Detect you will need to set up a corresponding “integration” in OpsGenie, which is described in the OpsGenie documentation topic Using API Integration. Assuming that you already have an OpsGenie account, you can add an integration for an existing team or create a new team of individual OpsGenie users that you specify (see the OpsGenie topic Create a Team).

Once you have a team, adding an integration involves the following steps:

1. In OpsGenie, go to the Integrations tab of your Team Dashboard.
2. Click the Add Integration button. The page will fill with a set of tiles, each representing an integration.
3. In the API tile, click the Add button. An Integration page will open (by default, the title will be your team name with “_API” appended).
4. Copy the API Key field. You’ll need to enter this key into the OpsGenie GenieKey field when you configure an OpsGenie notification channel in Kentik Detect.
5. Click the Save Integration button to save the Integration.

When you configure an OpsGenie channel in Kentik Detect and assign it to an Alert Policy (see Threshold Activation), notifications from thresholds that trigger alarms will appear in the OpsGenie Alert List. Click on an item in the list to see an Alert Detail (shown below), which contains effectively the same information that’s included in a JSON alert notification (see Sample Alert JSON).

The following setting is used only for PagerDuty alert notifications:

• PagerDuty Integration Key: A unique service identifier used by the PagerDuty Events API to trigger, acknowledge, and resolve incidents.

Before establishing a PagerDuty notification in Kentik Detect you will want to set up a corresponding service in PagerDuty (see PagerDuty’s Configuring Services and Integrations support page):

1. Configure at least one Service to hook notifications to. If desired, you can create multiple Services that go to different people depending on the type of alert.
2. On Integrations tab of the Service Details page for your service, click the New Integration button.
3. On the resulting Add an Integration page:
   - Enter a name for the integration (e.g. “Kentik”) in the Integration Name field.
   - For the Integration Type setting, be sure that the radio button is set to the “Select a tool” option, then choose “Kentik” from the drop-down list.
   - Click the Add Integration button, which takes you back to the Service Details page.
4. The Kentik integration will now be listed in the table of integrations. Copy the Kentik Integration Key. You’ll use this key as the value of the PagerDuty Integration Key setting in the Create/Edit Notification Channel dialog in Kentik.

Note: The data included in a PagerDuty alert notification from Kentik Detect is effectively the same as what’s included in a JSON alert notification (see Sample Alert JSON).

The following settings are used only for ServiceNow alert notifications:

• ServiceNow Instance Name: A unique identifier used by ServiceNow.
• ServiceNow Username: The username associated with the ServiceNow Instance.
• ServiceNow Password: The password associated with the ServiceNow Instance.

Before establishing a ServiceNow notification channel in Kentik Detect you will need to register for ServiceNow, log into your account, and request a ServiceNow “instance” (see the ServiceNow training topic Personal Developer Instances). Getting the information you need to configure ServiceNow in Kentik Detect involves the following steps:

1. In ServiceNow, choose Instance from the Manage menu.
2. Click the Request Instance button and specify a ServiceNow version for the instance.
3. The page will update to display the name, username, and password for the new instance. You’ll need to enter this information into the fields listed above when you configure a ServiceNow notification channel in Kentik Detect.

Once you configure an ServiceNow channel in Kentik Detect and assign it to an Alert Policy (see Threshold Activation), notifications from thresholds that trigger alarms will be listed on the ServiceNow Incidents page. Click on an item in the list to see the details page for an incident (shown below), which effectively contains the same information that’s included in a JSON alert notification (see Sample Alert JSON).

Slack channel alert notifications use only the Common Notification Settings in the Add Notification Channel or Edit Notification Channel dialog. In addition to settings in the dialog, however, setup of Slack alert notifications also involves configuration in Slack itself:

1. Before establishing a Slack notification in Kentik Detect, first create the channel in Slack on which you wish to receive notifications from Kentik Detect.
   Note: Confirm the desired Public/Private setting of the new Slack channel before continuing to step 2.
2. In the Kentik Detect dialog (either Add or Edit), select Slack as the notification type, enter a name for the channel, and then click the Save button. You will be taken to Slack to complete setup of the channel.
3. In Slack, you’ll be asked to allow the Kentik Alerts application to access your “team” (your company or organization).
4. Still in Slack, you’ll be asked to select the Slack channel that the Kentik Alerts application can post to. Choose the Slack channel that you created in step 1. The channel that you named in step 2 is now associated with that Slack channel.

If you created a Slack notification channel from within the Kentik portal but are not receiving notifications, troubleshoot as follows:

1. Contact your organization’s Slack administrator and have them verify:
   - your allowed applications;
   - the permissions that enable you to add integrations and applications.
2. In Kentik Detect, in the Notification Channel List on the Channels page (Alerting » Notification Channels), find the channel that you created (in step 2 of Slack Notification Settings) to receive Slack notifications from Kentik. Use the Remove button (trash icon) at the right of the row to remove the channel.
3. Redo steps 2 through 4 in the Slack Notification Settings topic.
4. If the issue persists, repeat step 2 above until you have removed all existing notification channels that use Slack. Then recreate each of the deleted channels (see steps 2 through 4 in Slack Notification Settings).

Problems with Slack notification can be caused by changing the Public/Private setting of a Slack channel in your organization’s Slack system after that channel has been assigned to a Kentik notification channel (e.g. integrating a channel that’s Public, then switching that channel to Private). If you need to change the setting, first remove the notification channel from Kentik, then modify the Slack channel, and then re-create the notification channel in Kentik.

The following settings are used only for System Log (syslog) alert notifications:

• Destination IP: IP address to which Kentik Detect notification messages will be posted.
• Port: Port on which syslog will listen for notifications.
• Transport Type: Protocol, either UDP or TCP
• Hostname: Name for the syslog to which messages will be posted.
• Severity: The severity level of the notification messages sent via this channel. Options include emergency, alert, critical, etc. as defined in RFC 5424.
• Facility: The facility code of the notification messages sent via this channel. Options include kernel, user, system, etc. as defined in RFC 5424.

When the notification type is set to System Log the Add Notification Channel or Edit Notification Channel dialog will also display a Syslog Security Information field stating the IP/CIDR from which notifications will be sent.

Note: The data included in a Syslog alert notification from Kentik Detect is effectively the same as what’s included in a JSON alert notification (see Sample Alert JSON).