In this article:

Contents Search
   

 

Flow Overview

This article covers the following topics related to flow and the configuration of routers to collect and export it:

Note: For information about configuring devices to send flow to Kentik Detect, see:
- Router Configuration for routers;
- Host Configuration for hosts.

 

 
 top

About Flow

Flow records are metadata (information) about each IP conversation (collection of related packets) that traverses a device such as a router, switch, or host. If a given device is configured to enable it, a flow record (data about a flow) can be collected in a cache and exported by sending it to a specified destination (e.g. Kentik Detect) at a specified interval. For example if IP 1.1.1.1 is sending packets to 2.2.2.2 through a flow-enabled device, information about that conversation can be collected in a flow record that includes the following basic flow fields:

Field Description
source IP Source IP address
dest IP Destination IP address
protocol IP protocol type (for example, TCP = 6; UDP = 17)
L4 source port TCP/UDP source port number or equivalent
L4 dest port TCP/UDP destination port number or equivalent
TCP flags (logical OR) Cumulative OR of TCP flags
input interface ID SNMP index of input interface
output interface ID SNMP index of output interface
byte count Total number of Layer 3 bytes in the packets of the flow
packet count Packets in the flow
ToS / DSCP value IP type of service (ToS)
src IP mask Source address prefix mask bits
dst IP mask Destination address prefix mask bits
next-hop IP IP address of next hop router

 

 
 top

Flow Protocols

Four primary protocols exist for flow data. Kentik Detect accepts all four protocols. The protocol used to send flow data to Kentik from a given device should be chosen based on what that router/switch supports and handles most efficiently. Where multiple protocols are supported, Kentik recommends them in the following order:

  • sFlow: designed by the sFlow.org consortium as a statistical monitoring tool for networks; configurable through SNMP.
  • IPFIX (Internet Protocol Flow Information Export): created by the Internet Engineering Task Force (IETF) as a universal standard for exported flow data.
  • NetFlow version 9: a Cisco flow protocol that is compatible with and nearly identical to IPFIX.
  • NetFlow version 5 (known as “JFlow” on Juniper devices): an earlier Cisco flow protocol.

The following table shows some of the features that are supported in the various protocols:

Features NetFlow v5 NetFlow v9 IPFIX sFlow
Basic flow fields
(see About Flow)
Yes Yes Yes Yes
Embedded sampling rate Yes No No Yes
IPv6 support No Yes Yes Yes
MAC address fields No Available on some platforms
using custom template
Available on some platforms
using custom template
Yes
VLAN ID fields No Available on some platforms
using custom template
Available on some platforms
using custom template
Yes
Includes payload sample No No No Yes

Note: Some routers and switches will not report layer-2 traffic; they only report flows as they traverse a layer-3/route decision. For additional information, consult your device vendor.

 

 
 top

Flow Sampling

While it’s possible to configure a device flow processing system to examine every flow, the norm is to “sample” flow instead. Flow sampling means exporting a flow record for only one in every X flow rather than for every flow. Device vendors use a variety of algorithms (random, consistent, etc.) for flow sampling; Kentik Detect works with all such algorithms in current use.

Regardless of the size of your operation, flow sampling is recommended for the following reasons:

  • Sampling reduces the device cycles required for processing and collection of flow.
  • Sampling reduces the network utilization (bandwidth) required for sending flow data (the required bandwidth is typically not significant).
  • Sampling makes it easier to see flow data in real time, because when flow is unsampled some routers hold the flow data for minutes or longer before sending it to the collector.

Sampling is especially important during unexpected high traffic volumes. During an attack or high PPS event, for example, a router that may otherwise be able to handle unsampled flow can be overwhelmed by the sheer volume of data and may stop processing flow. That can cause a lack of visibility at precisely the moment when it is most critical to be able to collect and analyze traffic data.

The ideal sampling rate is low enough capture critical information but high enough to efficiently handle peaks. Below are recommended sample rate tiers based on total device traffic (the sum of all inbound traffic on all interfaces):

Data volume Recommended sample rate
< 100 Mb/s 1 in 256
< 1 Gb/s 1 in 512
< 10 Gb/s 1 in 1024
< 25 Gb/s 1 in 2048
> 25 Gb/s 1 in 8192

Note: If you are prone to attacks that are 10x your normal traffic, or if you have greater than 5x bandwidth available for attacks to utilize, sample at 1 or 2 tiers higher than the recommendations shown above.

 

 
 top

Ingress and Egress

Depending on how a given device is configured, flow may be created by examining traffic at either of the following points:

  • Ingress — as traffic comes into an interface;
  • Egress — as traffic exits an interface.

It is recommended that you enable flow on all interfaces and configure all devices for ingress flow creation only (NetFlow was originally designed for this scenario but has since expanded to allow egress flow creation as well to handle special cases such as compression and VPN services).

Enabling ingress flow creation on all interfaces will give you a full picture of all traffic traversing the router. The Kentik Detect system will, for example, allow you to examine traffic that has left an interface by grouping all flows that were destined to that interface as they traversed the ingress from other interfaces.

Enabling both ingress and egress flow creation may result in the same traffic being counted twice. Kentik Detect allows you to verify that flows are not being double-counted, because whenever you view traffic for an individual interface Kentik Detect will report both the flow traffic and the SNMP traffic (use the device/interface tab and select traffic or use the Data Explorer and filter by interface). When the flow traffic on an interface (or set of interfaces) is compared to the SNMP recorded interface traffic the two metrics should be within 20 percent of one another.

Note: Kentik Detect does not currently remove duplicate flows resulting from enabling both ingress and egress flow creation (it will in the future).