Notifications

Notification channels are discussed in the following topics:

• About Notifications
• Notifications Page
• Manage Notification Channels
• Notification Dialogs UI
• Notification Settings

Notes:
- For general information on policy alerting, see Policy Alerts Overview.
- For information on settings for alert policies, see Alert Policies.
- For information on active or historical alerts, see Alerting.
- For information on mitigation for alerts, see Mitigations.

Notifications in Kentik are sent via notification channels, which allow you to set the means by which Kentik communicates with users about important events. Each notification channel combines a specific type of notification (e.g. email, PagerDuty, Slack, etc.; see Notification Types) with single or multiple targets such as a set of email recipients or a destination URL.

The notification channels created in your organization, which are managed from the Notifications Page, are used in a variety of portal contexts:

• Alert Policies: Notifications when traffic meets the conditions specified in a threshold of an alert policy (Alerting or DDoS Defense), resulting in the triggering of an alarm (see Threshold Notifications).
• Mitigation: Notifications when an automatic mitigation using a given mitigation method is triggered by conditions set in a threshold of an alert policy (see Notification Channels in General Method Settings).
• Synthetics: Notifications triggered when a test result meets the criteria that is specified when a test is added or edited; see Alerting and Notifications.
• Insights: Notifications from insights, which are included in the Notifications List but are subscribed to directly from an insight; see Notifications under Details Sidebar.

Note: In the UI of the Notifications page, notification channels associated with alerts are labeled with the deprecated term "Custom Insights."

The Notifications page is covered in the following topics:

• About the Notifications Page
• Notifications Page UI
• Notifications List
• Notifications Filters

Notification channels are managed from the Notifications page. To access the page, select Settings from the main menu, then Notifications (under Customizations).

The settings on this page determine how or to whom a notification will be communicated. They do not determine when or why a notification channel should be used. For more information about how notifications are used, see Alerts and Policies.

Note: Both Members and Administrators can access and view the Notifications page, but only Admin users can create or edit notifications.

The Notifications page is home to the Notifications list, which shows all current system notification channels. The page includes the following main elements:

• Show/Hide Filters (funnel icon): A button to the left of the Search bar that toggles display of the Filters pane.
• Search: A field that allows you to search existing notification channels using a text string.
• Add Notification (Administrators only): A button that allows you to create a notification channel.
• Notifications list: A table listing all notification channels currently in use by your organization (see Notifications List).
• Filters: A pane that allows you to filter the Notifications list based on multiple parameters (see Notifications Filters).

The Notifications list is a filterable table that lists the notification channels created in your organization. Click on the heading of any column (with the exception of Used By) to sort the list (ascending or descending).

The table includes the following columns:

• Name: The user-defined name assigned to the notification.
• Type: The type of notification. (e.g. Email, Slack, JSON).
• Used By: Shows you what locations in the portal are using the notification channel. If the channel is currently in use, click the contents of the cell to open the Notification Channel Usage Detail dialog, which provides details on each entity (policy, test, etc.) that uses the channel. The table in this dialog is the same as the table on the Notifications Used By Tab of the Notification Settings dialogs.
• Daily Digest: True (checkmark) or false (blank); shows if the notification channel is being used for a daily digest.
• Status: Status of the notification channel can either be enabled (green) or disabled (red).
• Edit (pencil icon): Edit the notification channel. Only visible to Admin users.
• Remove (trash icon): Remove the notification channel from the Kentik system. Only visible to Admin users.

Note: In addition to the notification channels used by alerting and synthetics, this table also shows channels subscribed to by insights, whose notifications are always via email (see Notifications under Details Sidebar).

The Filters pane enables you to narrow the items displayed in the Notifications list:

• The list will display notification channels that match all of the criteria that you specify in the sections of the Filters pane.
• Selecting no options in a section is equivalent to selecting all options in that section.
• To reset your selections, click Clear all at the top of the Filters pane.

The Filters pane includes the following sections:

• Status: Enabled or disabled notification channels.
• Insight Names: Click in the field to drop down a list of insight names from which you can select an insight to filter by. Each selected insight will appear in the field as a lozenge.
  Note: Alerts in this list are labeled with the deprecated term "Custom Insights."
• Insight Families: Click in the field to drop down a list of insight families (see Insights Families) from which you can select a family to filter by. Each selected insight family will appear in the field as a lozenge.
• Type: The type of notification (see Notification Types).

When filters are selected in multiple sections (Status, Insight Names, Insight Families, or Type), only notifications matching all selected filters are displayed. For example, if you select Status = Enabled and Type = both Email and Slack, then the Notifications list will include only enabled notifications that use either Email or Slack.

The following topics cover the basics of managing notification channels in Kentik:

• Add or Edit Notification
• Enable or Disable Notification

Notification channels are added or edited using one of the following dialogs:

• Add Notification: Used to create a new alert notification channel. Access by clicking the Add Notification button on the upper right of the Notifications page.
• Edit Notification: Used to modify an existing notification channel. Access by clicking the Edit icon (pencil) on a notification channel’s row.

To configure notification channels in these dialogs:

• Refer to the documentation for detailed information on the specific notification channel type you want to add or edit (see Notification Types).
• Follow any preliminary steps external to Kentik that are required to set up the notification channel.
• Set the Common Notification Settings.
• Set any additional settings specific to the notification channel type (refer to the individual type settings in Notification Settings).
• Follow any additional steps external to Kentik that are required to complete setting up the notification channel.

Note: Notification channels from insights are managed separately; see Notifications under Details Sidebar.

The option to enable or disable a notification channel is displayed during the creation of the channel. The default state is Enabled. The current state is displayed in the Status column of the Notifications List.

To disable an enabled notification channel, or vice versa:

1. On the Notifications page, find the row for the notification.
2. Click the Edit icon at the right of the row.
3. In the resulting Edit Notification dialog, and use the Status switch toward the upper right to enable or disable the notification.

Note: When you disable a notification channel, any alerts or insights that use that channel will no longer be sent.

The Add Notification and Edit Notification dialogs share the same layout and the following common UI elements:

• Close: Click the X in the upper right corner to close the dialog. All elements will be restored to their values at the time the dialog was opened.
• Tab selector: Select which tab is displayed:
  - Settings tab: Contains the Name, Status, and Type fields (see Common Notification Settings) as well as other settings that vary by notification channel type.
  - Preview tab (Edit Notification dialog only): Contains the Preview field and a Test button (see Notifications Preview Tab).
  - Used By tab (Edit Notification dialog only): Contains a list of the alerts and/or insights that use this notification (see Notifications Used By Tab).
• Cancel: Click this button to cancel the add or edit operation and exit the dialog. All elements will be restored to their values at the time the dialog was opened.
• Add Notification (Add Notification dialog only): Click this button to save your settings for the new channel and exit the dialog.
• Save (Edit Notification dialog only): Click this button to save your changes to the notification channel settings and exit the dialog.

The settings of the Add Notification and Edit Notification dialogs depend on the type of notification channel you'd like to configure. These settings are covered in the following topics:

• Notification Types
• Common Notification Settings
• Custom Webhook Notification Settings
• Email Notification Settings
• JSON Notification Settings
• Microsoft Teams Settings
• OpsGenie Notification Settings
• PagerDuty Notification Settings
• ServiceNow Notification Settings
• Slack Notification Settings
• Splunk Notification Settings
• Syslog Notification Settings
• VictorOps Notification Settings
• xMatters Notification Settings

Kentik supports alert notification channels using the notification systems listed below. The settings for each type are covered in the listed topics:

• Custom Webhook: See Custom Webhook Notification Settings.
• Email: See Email Notification Settings.
• JSON: See JSON Notification Settings.
• Microsoft Teams: See Microsoft Teams Settings.
• OpsGenie: See OpsGenie Notification Settings.
• PagerDuty: See PagerDuty Notification Settings.
• ServiceNow: See ServiceNow Notification Settings.
• Slack: See Slack Notification Settings.
• Splunk: See Splunk Notification Settings.
• System Log (syslog): See Syslog Notification Settings.
• VictorOps: See VictorOps Notification Settings.
• xMatters: See xMatters Notification Settings.

Note: Insight notifications are currently supported via email channels only (see Notifications under Details Sidebar).

The settings below are common to the Settings and Preview tabs for every notification channel type.

The following common settings are present on the Settings tab:

• Name: A user-assigned name for the notification channel.
• Status: A toggle switch that determines whether the notification channel is enabled (available for use) or disabled.
• Type: The type of the notification channel. The remaining fields of the dialog vary depending on the type:
  - Add dialog: A set of buttons from which you can choose one of the types listed in Notification Types.
  - Edit dialog: An inactive button that serves as an indicator of the notification channel's type.

The Preview tab, which is accessible only in the Add Notification dialog, includes the following common settings:

• Preview: A locked field that displays the HTML markup for the notification channel.
• Test: Click this button to receive a test notification via the selected notification channel type.

The Used By tab contains a sortable table that shows all of the entities (alerts, tests, insights, etc.) that currently use this notification channel. The table includes the following columns and actions for each alert:

• Where: The insights, insight families, mitigation methods, policies, and synthetic tests that use this channel for notifications.
• ID: The ID of the entity listed in the Where column. An ID may be numeric (e.g. for policies and tests) or a string (e.g. for insights).
• Name: The name of the entity listed in the Where column.
• Link: A link that takes you away from the Notifications page to the page where the entity listed in the Where column can be managed.
  Note: If the entity is a custom insight then the existing browser tab is unchanged and the link is opened in a new browser tab.
• Remove (trash icon): A button that opens a confirmation dialog that allows you to stop notifications via this channel from the entity listed in the Where column.

Custom webhook notification channels are Kentik’s most powerful and flexible way of integrating Kentik notifications with third party or custom output channels. They provide a programmatic approach to sending notifications to an API endpoint.

The following fields are used to create Custom Webhook notification channels:

• URL: The address to which Kentik should send notifications.
• Custom Headers: A set of one or more text fields, each of which is used to specify one custom HTTP header to include in notifications from this channel. The following types of headers are supported:
  - Authorization
  - Content-Type
  - Any HTTP headers that start with an x- prefix
• Add: A button that adds another field for a custom header.
• Custom Template: A field in which to describe the notification channel's payload using Go-template syntax (see Using Go Templates). This allows you to specify the contents of the API request. For more information, see Using Custom Webhook Templating.
• Uglify JSON: A switch to enable or disable the removal of unnecessary whitespace from a JSON payload. The default is Off.

Kentik relies on the templating syntax of the Go programming language for creation of custom webhook templates, but we’ve added convenient functions and variables that allow you to refer to the properties and attributes of the notifications.

• For more information on how to use custom webhooks, see:
• Kentik Custom Notification Templates
• Go Template Documentation
• Learn Go Template Syntax

In addition to the elements detailed in Common Notification Settings, the following Settings tab elements are used for email notification channels:

• Email Addresses: A set of one or more text fields, each of which is used for the address of one email recipient.
• Add: A button that adds an email address field.

The image below shows a representative alert notification email from Kentik.

Note: A notification email generated by an alert policy contains links that enable you to view the traffic that triggered the alarm, either in the Alert Details Page or in the alert policy's designated "policy dashboard" (see General Policy Settings).

In addition to the elements detailed in Common Notification Settings, the following Settings tab field is present when the notification channel type is JSON:

• URL: The URL to which Kentik should post JSON notifications for this channel.

JSON notification channels enable you to integrate Kentik with third-party monitoring systems so that events detected by Kentik (e.g. an anomaly detected by our Alerting system) can trigger external actions, which may include network configurations, DDoS attack mitigations, or other remedies. The JSON payload is posted to the specified Webhook URL, where it can be parsed and processed for any desired purpose. For an example of the JSON payload that will be posted, see Sample Alert JSON.

JSON may be a good option for your notification channels if you prefer not to define the syntax (unlike custom webhooks). It provides a hard-coded format of the notification payload without customization, enabling users to rely on a stable and defined payload.

You can secure your receipt of JSON alert notifications from Kentik using any of the following complementary methods:

• Filter by IP: Filter inbound POST requests (using iptables on the web-service server or any firewall in the path, or in the code of the web service itself) to only those from IPs in the netblock 209.50.158.0/23.
• Filter by HTTP header: Filter inbound POST requests to only those with the following header: "User-Agent:KentikAlerting"
• Use a query argument: If you use HTTPS for the Webhook, include a query argument known only to your responding web-service.

If problems arise when testing JSON notification channels, ensure that your web server is available to accept HTTP requests and also that it is accessible via a public URL. Kentik suggests trying the methods below to debug any issues.

• RequestBin: Use the free web-service RequestBin (https://requestbin.com) to collect HTTP requests in bins. You can examine these to see the contents of the request.
• ngrok: If you have a development web server on your local host that accepts HTTP-POST, use ngrok (https://ngrok.com/) to get a unique public URL that is directly connected to your development machine.

The following example (with placeholders highlighted) illustrates the JSON that would typically be sent as a notification for an insight (see Insight Notification Settings).

{
  "CompanyID":####,,
  "CurrentState": "n/a",
  "Description": "Insight for Through Site Comparison",
  "Dimensions": {},
  "EndTime": "2022-11-02 13:50:00 UTC",
  "InsightDataSourceType": "ksql",
  "InsightID": "k654321097",
  "InsightName": "operate.site.bpsThroughWeekOverWeek",
  "InsightPlainDescription": "Foobar Amsterdam AMS13 had 22% more through traffic (+139 Kbits/s) this week compared to last week.",
  "IsActive": true,
  "Links": {
    "InsightDetailsURL": "https://portal.kentik.com/v4/operate/insights/k611587433",
    "InsightsMainURL": "https://portal.kentik.com/v4/operate/insights",
    "InsightsSeverityURL": "https://portal.kentik.com/v4/operate/insights?severities=notice"
  },
  "Metrics": {},
  "PreviousState": "n/a",
  "StartTime": "2022-11-02 11:50:00 UTC",
  "Type": "EVENT_VIEW_TYPE_INSIGHT",
  "issue": [],
  "statistic": {}
}

The following example (with placeholders highlighted) illustrates the JSON that would typically be sent in response to a change in the state of an alert (see Threshold Notifications).

{
  "EventType":"ALARM_STATE_CHANGE",
  "CompanyID":####,
  "MitigationID":0,
  "AlarmID":#####,
  "AlarmState":"ALARM",
  "PolicyID":####,
  "ThresholdID":####,
  "ActivateSeverity":"major",
  "AlarmStart":"2017-05-08T18:17:18Z",
  "AlarmEnd":"0001-01-01T00:00:00Z",
  "LastActivate":"2017-05-08T18:17:18Z",
  "AlertPolicyName":"TEST_ALWAYS_FIRE",
  "AlarmStateOld":"",
  "AlertKey":[
    {
      "DimensionName":"IP_dst",
      "DimensionValue":"###.###.###.#"
    }
  ],
  "AlertValue":{
    "Unit":"Packets/s",
    "Value":21760
  },
  "AlertBaseline":{
    "Unit":"Packets/s",
    "Value":1000
  }
  "AlertBaselineSource": "ACT_BASELINE_MISSING_DEFAULT_INSTEAD_OF_LOWEST"
}

The following table explains the elements of the notification JSON:

The following table shows the source of the information returned in the AlertBaselineSource element of the notification JSON:

In addition to the elements detailed in Common Notification Settings, the following Settings tab field is present when the notification channel type is Microsoft Teams:

• URL: The URL to which Kentik should post Microsoft Teams notifications for this channel.

To use a Microsoft Teams notification channel in Kentik:

1. Set up an incoming webhook in Microsoft Teams as described in the Microsoft documentation topic Create an Incoming Webhook.
2. Note the unique webhook URL in step 5.
3. In Kentik, enter the webhook URL into the URL field described above.

Kentik supports the following notification sources for use with Microsoft Teams:

• Synthetic tests (failure or recovery)
• DDoS alert
• DDoS mitigation methods

In addition to the elements detailed in Common Notification Settings, the following Settings tab field is present when the notification channel type is OpsGenie:

• Token: A unique API token used by the OpsGenie Web API.

To use an OpsGenie notification channel in Kentik:

1. Set up a corresponding “integration” in OpsGenie as described in the OpsGenie documentation topic Create an API Integration.
2. Once you have an OpsGenie account, you can add integrations for an existing team or create a new team of individual OpsGenie users.
3. Once you have a team, add an integration using the steps provided in OpsGenie’s Using API Integration topic.
4. In Kentik, copy the API key provided in OpGenie's step 5 and paste it into the Token field above.

When OpsGenie notification channels in Kentik are assigned to alert policies (see Alerts and Policies), a notification from a triggered threshold in a policy will appear in the alert list in OpsGenie. Click on an item in the list to see an Alert Detail that contains effectively the same information that's included in a JSON alert notification (see Sample Alert JSON).

The following setting is used only for PagerDuty notification channels:

• Token: A unique service identifier used by the PagerDuty Events API to trigger, acknowledge, and resolve incidents.

To establish a PagerDuty notification channel in Kentik:

1. Set up a corresponding service in PagerDuty (see PagerDuty's Configuring Services and Integrations support page).
2. In Kentik, put the integration key created by this process into the Token field above.

Note: The data included in a PagerDuty alert notification from Kentik is similar to a JSON alert notification (see Sample Alert JSON).

In addition to the elements detailed in Common Notification Settings, the following Settings tab fields are present when the notification channel type is ServiceNow:

• Instance: The URL of the ServiceNow Instance.
• Username: The username associated with the ServiceNow Instance.
• Token: The password associated with the ServiceNow Instance.

To establish a ServiceNow notification channel in Kentik:

1. Log into your ServiceNow Developer Site.
2. Create a ServiceNow “instance” as described in the ServiceNow training topic Personal Developer Instances.
3. In Kentik, put the instance name, username, and token generated by ServiceNow into the corresponding fields of the Settings tab.

When a ServiceNow notification channel created in Kentik is assigned to an alert policy, notifications triggered by that policy's thresholds will be listed on the Incidents page in ServiceNow. Click an item in the list to view the details of that incident. The data included in a ServiceNow alert notification from Kentik is similar to a JSON alert notification (see Sample Alert JSON).

In addition to the elements detailed in Common Notification Settings, the following Settings tab field is present when the notification channel type is Slack:

• URL: The URL to which Kentik should post Slack notifications.

To establish a Slack Notification channel, you need to set up a webhook in Slack (see Sending messages using Incoming Webhooks). Put the webhook URL generated in Slack into the URL field described above.

If you created a Slack notification channel from within the Kentik portal but are not receiving notifications, troubleshoot as follows:

1. Verify your allowed applications in Slack.
2. Verify your permissions in Slack.
3. In Kentik, in the Notifications List on the Notifications page (Settings » Notifications), find the channel that you created to receive Slack notifications from Kentik. Use the Remove button (trash icon) at the right of the row to remove the channel.
4. If there are existing Slack notifications within Kentik that function as expected, compare the settings of those Slack channels for differences (for example, public vs. private settings).

Problems with Slack notification can be caused by changing the Public/Private setting of a Slack channel in your organization’s Slack system after that channel has been assigned to a Kentik notification channel (e.g. integrating a channel that’s Public, then switching that channel to Private). If you need to change the setting, first remove the notification channel from Kentik, then modify the Slack channel, and then re-create the notification channel in Kentik.

In addition to the elements detailed in Common Notification Settings, the following Settings tab fields are present when the notification channel type is Splunk:

• URL: The URL to which Kentik should post Splunk notifications for this channel.
• Token: The HTTP Event Collector Token provided by Splunk.

To establish a Splunk notification channel in Kentik:

1. In Splunk, enable and configure the HTTP Event Collector (HEC) according to your specific type of Splunk software (see the Splunk documentation topic Set up and use HTTP Event Collector in Splunk Web).
2. Create an event collector token and copy its value. In Kentik, put this value into the Token field described above.
3. Determine the specific URL for your type of Splunk software as described in Send data to HTTP Event Collector. In Kentik, put this URL into the URL field described above.

In addition to the elements detailed in Common Notification Settings, the following Settings tab fields are present when the notification channel type is System Log (syslog):

• Host: IP address to which notification messages will be posted.
• Port: Port on which syslog will listen for notifications.
• Network Protocol: Protocol to send data, either UDP or TCP
• Syslog Hostname: Name for the syslog to which messages will be posted.
• Severity: The severity level of the notifications sent via this channel. Options include emergency, alert, critical, etc. as defined in RFC 5424.
• Facility: The facility code of the notifications sent via this channel. Options include kernel, user, system, etc. as defined in RFC 5424.

Note: The data included in a Syslog alert notification from Kentik is similar to a JSON alert notification (see Sample Alert JSON).

In addition to the elements detailed in Common Notification Settings, the following Settings tab field is present when the notification channel type is VictorOps (now known as Splunk On-Call):

• URL: The URL to which Kentik should post VictorOps notifications.

To establish a VictorOps notification channel in Kentik:

1. In VictorOps, enable a REST Endpoint Integration and set up a REST Endpoint Integration Routing Key, as described in the documentation topic REST Endpoint Integration Guide - Splunk On-Call.
2. Use the generated routing key to create a VictorOps webhook URL.
3. In Kentik, enter the webhook URL into the URL field described above.

In addition to the elements detailed in Common Notification Settings, the following Settings tab field is present when the notification channel type is xMatters:

• URL: The URL to which Kentik should post xMatters notifications for this channel.

To establish an xMatters notification channel in Kentik:

1. Set up an incoming webhook in xMatters, as described in the xMatters documentation topic Webhooks.
2. In Kentik, enter the webhook URL into the URL field described above