CH logo® Knowledge Base
Contents Search
   

 

Dimensions and Metrics

The features and use of the Data Explorer in the Kentik Detect portal are discussed in the following topics:

Note: In addition to being used for settings in the Kentik Detect portal, dimensions and metrics are also used for the Query API.
- In addition to the standard dimensions and metrics included with Kentik Detect, users can create custom dimensions (see Custom Dimensions).

 

 
 top

About Dimensions

Dimensions in Kentik Detect represent specific data about flow (see Flow Overview). This data is either taken directly from flow records (e.g. NetFlow, sFlow), incorporated from correlated sources (e.g. GeoIP or threat feeds), or derived by Kentik from some combination of the two. Each dimension represents a column in the main tables of the KDE (see Main Table Schema).

Dimensions are primarily used in the following contexts in the Kentik Detect portal and the Query API:

  • Group-by dimensions: Selected in the Query pane of the sidebar (e.g. in Data Explorer).
  • Filters: Selected in the Ad-Hoc Filter Groups section of the Filtering Options dialog accessed from the Filtering pane of the sidebar.

The following table shows more specifically the various places in the Kentik Detect portal where dimensions are specified and used:

Portal section Group-by Filters
Dashboards - Add Panel or Edit Panel dialog » Edit Query in Data Explorer button » sidebar Query pane.
- See Panel Dialogs.
- Sidebar Filtering pane » Filtering Options dialog » Ad-Hoc Filter Groups pane.
- See Ad-Hoc Filter Interface.
Data Explorer - Sidebar Query pane » Group-by Dimensions dialog.
- See Query Dimension Selectors.
- Sidebar Filtering pane » Filtering Options dialog » Ad-Hoc Filter Groups pane
- See Ad-Hoc Filter Interface.
Alerting » Policies Add Policy or Edit Policy dialog » Dataset tab » Data Funneling pane » Dimensions. Add Policy or Edit Policy dialog » Dataset tab » Data Funneling pane » Filters.
Analytics » Peering, Add Dataset N.A. - Add Dataset dialog » Filters pane.
- See Filtering Pane Settings.
Analytics » Peering, Analysis View N.A. - Sidebar » Filters pane.
- See Filtering Pane Settings.
Analytics » Route Traffic N.A. - Sidebar Filtering pane » Filtering Options dialog » Ad-Hoc Filter Groups pane
- See Ad-Hoc Filter Interface.
Analytics » Capacity Sidebar Options pane » Group-by Dimensions dialog (includes only subset of dimensions). N.A.
Admin » User N.A. - Add User or Edit User dialog » User Specific Filters pane.
- See User Admin Dialogs.
Admin » Saved Filters N.A. - Add Saved Filter or Edit Saved Filter dialog » Ad-Hoc Filter Groups pane.
- See Saved Filter Admin Dialogs.

 

 
 top

Dimension Categories

In general, dimensions are available in dimension and filter selectors for queries involving traffic from all types of devices (both routers and hosts), though some dimensions apply only to traffic from host agents such as kprobe (see About kprobe).

Dimensions, which represent or are derived from columns in KDE (see Main Table Schema) fall into the following functional categories:

Category Description Requires host agent (kprobe)
Time Used to specify time ranges. No
IP IP addresses (Ipv4 or Ipv6) of flow, as well as protocol (e.g. TCP or UDP), TCP flags, and ToS. No
Device & Port Information related to devices including interface names and descriptions, port IDs, etc. No
Geo Country codes, region names, and city names for the flow’s source and destination. No
BGP Routing information including source and destination AS, AS path, AS names, community, prefixes, and hops. No
LAN Source and destination local area network IDs and MAC addresses. No
Tags Matches to user-specified tags (see About Flow Tags). No
Query (KDE) Used to specify dataseries (Full or Fast; see Resolution Overview), and other query-related data. No
DNS/WWW Data related to DNS lookup and HTTP, including domain name, referrer, status, etc.). Yes
Ultimate Exit Information about where the traffic leaves the network (device, interface, site, etc.); see Using Ultimate Exit. No
Interface Classification Information about the interface though which a flow enters and leaves a device; see Interface Classification. No
Provider Classification Information (derived during interface classification) about the provider via which traffic from a given externally facing interface reaches the Internet; see Provider Classification. No
Threat Feed Information about source and destination hosts and IPs that have been identified as a security threat by Spamhaus (updated daily). No

Notes:
- For more detailed information about dimensions requiring a host agent, see Host Traffic Dimensions.
- For lists of the specific dimensions in each of the above categories, see Main Table Schema.
- In addition to the dimension categories listed above, KDE main tables also include some columns that store metrics (e.g. NPM), which are covered in General Metrics and Host Traffic Metrics.

 

 
 top

Host Traffic Dimensions

While most dimensions are available (for both group-by and filters) on traffic from all devices, the following dimensions are available only for traffic from a host agent such as kprobe (see About kprobe):

  • DNS Query: A query from a DNS resolver to a DNS name server that translates a user-friendly domain name (e.g. www.domain_name.com) to a numeric IP address, either 32-bit IPv4 (93.184.216.119) or 128-bit IPv6 (2606:2800:220:6d:26bf:1447:1097:aa7).
  • DNS Query Type: The resource record type requested by the DNS query. For a list of record types, see https://en.wikipedia.org/wiki/List_of_DNS_record_types.
  • DNS Response: The response from a DNS server to a DNS query. DNS responses are comprised of resource records (RRs). kprobe collects information from the following RRs:
    - A: IPv4 address for given host.
    - AAAA: IPv6 address for given host.
    - CNAME: A domain name that must be queried to resolve the original DNS query.
    - PTR: Used to look up a domain name based on an IP address.
    - MX: A mail exchange server for a DNS domain name.
    - NS: An authoritative name server for given host.
    - TXT: A non-formatted text string typically used by Sender Policy Framework (SPF) to prevent the sending of emails using a fake identity.
  • DNS Return Code: Status code returned from a DNS query (see https://www.iana.org/assignments/dns-parameters/dns-parameters.xhtml#dns-parameters-6).
  • HTTP Host Header: A mandatory HTTP header field that identifies the domain name of the server.
  • HTTP Referrer: A non-mandatory HTTP header field that identifies the address from which a destination webpage is requested. Logging referrers allows websites to keep track of how users arrive at a page.
  • HTTP Return Code: Status codes defined in the following document: https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html
  • HTTP User Agent: A non-mandatory HTTP header field that identifies the client that submitted a request. User agent information, such as operating system and browser software, helps websites determine how content will be displayed.
  • HTTP URL: The filename portion of a path to a web resource, with query string (if any).

For additional information on querying using host data, refer to Query Pane Settings. Options include matching substrings in the values of group-by dimensions (see DNS/WWW Extract Function).

 

 
 top

Dimensions and Columns

The dimensions that you can group by and filter by are based on columns in the main tables of KDE (see Main Table Schema). Some columns are native (actually stored in the backend) while others are virtual (derived from other information). The tables in the following topics show the correspondence between dimensions and columns:

Note: For fastest results, when possible use dimensions based on native columns over those based on virtual columns.

 

 
 top  |  section

Source and Destination Dimensions

Dimensions listed in the Source and/or Destination sections of the Dimension selector:

Dimension name (portal) Source KDE column Destination KDE column Column type
Custom Geo kt_src_market kt_dst_market Native
Country src_geo dst_geo Native
Region src_geo_region dst_geo_region Native
City src_geo_city dst_geo_city Native
AS Number src_as dst_as Native
AS Name i_src_as_name i_dst_as_name Virtual
Flow Tag src_flow_tags dst_flow_tags Native
IP Port l4_src_port l4_dst_port Virtual
MAC Address src_eth_mac dst_eth_mac Native
VLAN vlan_in vlan_out Native
IP/CIDR inet_src_addr inet_dst_addr Native
Interface ID input_port output_port Native
Interface Name i_input_interface_description i_output_interface_description Virtual
Interface Description i_input_snmp_alias i_output_snmp_alias Virtual
Interface Capacity (Mbps) i_input_interface_speed i_output_interface_speed Virtual
Connectivity Type i_src_connect_type_name i_dst_connect_type_name Virtual
Network Boundary i_src_network_bndry_name i_dst_network_bndry_name Virtual
Provider i_src_provider_classification i_dst_provider_classification Virtual
Route Prefix inet_src_route_prefix inet_dst_route_prefix Native
Route LEN src_route_length dst_route_length Native
BGP AS_PATH src_bgp_aspath dst_bgp_aspath Native
BGP Community src_bgp_community dst_bgp_community Native
Next Hop IP/CIDR inet_src_next_hop inet_dst_next_hop Native
Next Hop AS Number src_nexthop_as dst_nexthop_as Native
Next Hop AS Name i_src_nexthop_as_name i_dst_nexthop_as_name Virtual
2nd BGP_HOP AS Number src_second_asn dst_second_asn Native
2nd BGP_HOP AS Name i_src_second_asn_name i_dst_second_asn_name Virtual
3rd BGP_HOP AS Number src_third_asn dst_third_asn Native
3rd BGP_HOP AS Name i_src_third_asn_name i_dst_third_asn_name Virtual
BGP Ultimate Exit Site N.A. i_ult_exit_site Virtual
BGP Ultimate Exit Site Country N.A. i_ult_exit_site_country Virtual
BGP Ultimate Exit Device Name N.A. i_ult_exit_device_name Virtual
BGP Ultimate Exit Interface ID N.A. ult_exit_port Native
BGP Ultimate Exit Interface Name N.A. i_ult_exit_interface_description Virtual
BGP Ultimate Exit Interface Description N.A. i_ult_exit_snmp_alias Virtual
BGP Ultimate Exit Connectivity Type N.A. i_ult_exit_connect_type_name Virtual
BGP Ultimate Exit Network Boundary N.A. i_ult_exit_network_bndry_name Virtual
BGP Ultimate Exit Provider N.A. i_ult_provider_classifcation Virtual
Bot Net CC src_threat_bnetcc dst_threat_bnetcc Native
Threat List Host src_threat_host dst_threat_host Native
CDN src_cdn dst_cdn Native
Traffic Origination i_trf_origination N.A. Virtual
Traffic Termination N.A. i_trf_termination Virtual

 

 
 top  |  section

Full Dimensions

Dimensions listed in the Full section of the Dimension selector:

Dimension name (portal) KDE column Column type
TCP Flags tcp_flags Native
TCP Flags (raw) tcp_flags_raw Native
Protocol protocol Native
Protocol Name i_protocol_name Virtual
INET Family inet_family Native
Device ID i_device_id Virtual
Device Name i_device_name Virtual
Device Type i_device_type Virtual
Site i_device_site_name Virtual
Site Country i_device_site_country Virtual
Device Labels i_device_label Virtual
TOS/Difserv tos Native
Per-flow bytes (recorded inbound) in_bytes Native
Per-flow packets (recorded inbound) in_pkts Native
Per-flow bytes (recorded outbound) out_bytes Native
Per-flow packets (recorded outbound) out_pkts Native
Packet Size sampledpktsize Native
Packet Size (nearest 100) sampledpktsize_100 Native
Sampling Rate * 100 sample_rate Native
Traffic Profile i_trf_profle Virtual
OTT Service service_name Native

 

 
 top  |  section

Layer 3+ Dimensions

Dimensions listed in the Layer 3+ section of the Dimension selector:

Dimension name (portal) KDE column Column type
Host Direction i_host_direction Virtual
DNS Query kflow_dns_query Native
DNS Query Type kflow_dns_query_type Native
DNS Return Code kflow_dns_ret_code Native
DNS Response kflow_dns_response Native
HTTP URL kflow_http_url Native
HTTP Status kflow_http_status Native
HTTP Referrer kflow_http_referer Native
HTTP User Agent kflow_http_ua Native
HTTP Host Header kflow_http_host Native
TCP Retransmits retransmitted_out_pkts Native
Repeated TCP Retransmits repeated_retransmits Native
TCP Receive Window receive_window Native
TCP Zero Windows zero_windows Native
TCP Client Latency (ms) client_nw_latency_ms Native
TCP Server Latency (ms) server_nw_latency_ms Native
Application Latency (ms) appl_latency_ms Native
First Payload Exchange Latency (ms) fpex_latency_ms Native
Connection ID connection_id Native

 

 
 top

About Metrics

A metric is a combination of a unit (e.g. a bit) with a method of calculation (e.g. average per second) to create a quantifiable measurement (average bits/second). In Kentik Detect, metrics represent measurements made on flows, which are used for counts, rankings (e.g. in a top-X list), and thresholds (e.g. in alerting). The following table shows the main places in the Kentik Detect portal where metrics are specified and used:

Portal location Primary Metric Secondary Metric
Library » Dashboards: Add View Panel or Edit View Panel dialog Query tab » Query pane, either:
- Metrics field
- Customize Metrics » Metrics dialog (see Metrics Dialog).
Query tab » Query pane » Customize Metrics » Metrics dialog.
Data Explorer: Sidebar Query pane. Sidebar » Query pane, either:
- Metrics drop-down (see Query Basic Options).
- Customize Metrics » Metrics dialog.
Sidebar » Query pane » Customize Metrics » Metrics dialog.
Alerting » Policies: Add Policy or Edit Policy dialog Dataset tab » Data Funneling pane » Primary Metric drop-down. Dataset tab » Data Funneling pane » Secondary Metric field.

 

 
 top

General Metrics

The metrics available in a given setting (e.g. drop-down Metric list in Query pane of Data Explorer sidebar) vary depending on the device type. If the traffic being queried comes only from routers then metric options include only the following (for descriptions of the KDE columns from which metrics are derived, see Metrics Columns):

Metric in portal Derived from these main table column(s)
Bits per second in_bytes, out_bytes
Packets per second in_pkts, out_pkts
Count (flows per second) Based on rows per table
Unique IPs, source and destination inet_src_addr, inet_dst_addr
Unique prefixes, source and destination inet_src_route_prefix, inet_dst_route_prefix
Unique ASNs, source and destination src_as, dst_as
Unique next-hop ASN (destination) i_dst_nexthop_as_name
Unique countries, source and destination src_geo, dst_geo
Unique regions, source and destination src_geo_region, dst_geo_region
Unique cities, source and destination src_geo_city, dst_geo_city
Sampling rate, max and average sample_rate

Note: Metrics whose name includes “unique” involve evaluating all instances across all queried devices within each individual time slice covered by the query’s time range (see Table Time-slicing) but not between time slices.

If the traffic being queried includes traffic from any hosts (i.e. Kentik-registered devices representing a kprobe host agent), the following additional metric options are available (for descriptions see Host Traffic Metrics):

Metric in portal Derived from these main table column(s)
Retransmits, per second and % tcp_retransmit (and both_pkts for %)
Out-of-order packets, per second and % ooorder_in_pkts, ooorder_out_pkts (and both_pkts for %)
Fragments, per second and % fragments (and both_pkts for %)
Network latency per client/server/application (ms) client_nw_latency_ms, server_nw_latency_ms, appl_latency_ms

 

 
 top

Host Traffic Metrics

In addition to the metrics described in General Metrics, Kentik Detect uses data from kprobe (see About kprobe) to make available the following metrics on traffic from hosts:

  • Retransmits/second: Packets per second that are re-sent from source to destination; applies only to traffic using a reliable transport protocol such as TCP. Retransmission indicates network delivery issues that can degrade application/service performance.
  • % Retransmits: Resent packets (see above) as a percentage of all packets sent. This measure reveals aggregate efficiency in the full context of expected total capacity.
  • Out of order/second: Packets per second that arrived at the destination out of sequence. A high value indicates potential variability in delivery paths and is particularly detrimental to latency-sensitive traffic such as real-time audio or video.
  • % Out of order: Out of order packets (see above) as a percentage of all packets sent.
  • Fragments/second: Packets per second that have been split into smaller packets for delivery across the network. Fragmentation and reassembly can increase CPU load and also cause retransmits when fragments arrive out-of-order.
  • % Fragments: Fragmented packets (see above) as a percentage of all packets sent.
  • RTT/2 client latency (derived): One-way network latency as measured from the client perspective (in a client-server relationship). High values indicate latency problems in the network or at the server end of the flow.
  • RTT/2 server latency (derived): One-way network latency as measured from the server perspective (in a client-server relationship). High values indicate latency problems in the network or at the client end of the flow.
  • RTT/2 application latency (derived): A measure of one-way network latency that is derived by examining request/response pairs at the application layer (i.e. HTTP GET vs. first response). While this measure can be a reasonable proxy for end-user experience of application response, it only works for application protocols that have clear request/response pairings.
  • First Payload Exchange Latency: Used to measure application response time, particularly when the protocol isn’t understood or can’t be decoded (e.g. HTTPS, SQL, etc.). The time, which excludes TCP setup, starts with the first packet sent (typically the request) and stops with the first packet returned (typically the response).

Notes:
- If you’d like to limit your results to series in which a given metric is above a minimum value, you can set a threshold in the Metrics Dialog (see Set Threshold in Metrics Dialog UI).
- For greater clarity when querying latency, include a filter limiting the query to traffic using the TCP protocol (source, destination, or full).
- Sankey diagrams are not currently accurate for displaying latency or percent metrics. This is scheduled to be addressed in 2018 Q1.
- The KDE columns corresponding to the metrics listed above are shown in NPM Columns.

 

In this article: