In this article:

Contents Search
   

 

NetFlow Proxy Agent

The following topics cover the use of Kentik’s NetFlow proxy agent, chfagent, to enable encrypted local flow from your organization’s routers and switches to Kentik Detect:

 

 
 top

Proxy Agent Overview

The Kentik NetFlow proxy agent is covered in the following topics:

 

 
 top  |  section

About the Proxy Agent

Kentik allows customers to send the network data collected from their devices to Kentik Detect securely using a downloadable software agent called “chfagent.” Among other functions, chfagent acts as a NetFlow proxy agent that enables the local encryption of flow records (NetFlow v5/v9, IPFIX, and sFlow) before those records are forwarded to Kentik.

The machine running chfagent isn’t actually handling traffic directly, but rather allows flow and SNMP from routers to be locally collected and encrypted for transport to Kentik Detect. A single instance of the chfagent executable can redirect flow for multiple routers and switches, handling rate limiting and resampling as well as encryption. Multiple servers across the network can run chfagent to distribute traffic and load.

The steps involved in downloading, installation, and configuration of chfagent is described in the following topics below:

Note: For assistance with any aspect of the setup process, please contact support@kentik.com.

 

 
 top  |  section

Kentik Detect Traffic Flows

The diagram below shows an overview of traffic flows between the Kentik NetFlow proxy agent, installed in a customer backbone or IT facility, and Kentik Detect:

 

 
 top  |  section

Connections and Behavior

The following points describe agent behavior and the connection between customer devices, the Kentik proxy agent, and Kentik Detect:

  • chfagent transports all traffic received from customer devices securely to Kentik Detect.
  • The Kentik Data Engine and the Kentik Detect portal handle data forwarded via chfagent identically to data received directly from customer devices.
  • chfagent connects to any customer devices sending NetFlow telemetry to Kentik Detect, and also to Kentik Detect itself to send the encrypted data as well as to receive configuration information.
  • In order to send traffic to Kentik Detect, chfagent will build, for each NetFlow-sending device, two HTTPS sessions, one for flow traffic and the other for SNMP. All traffic is sent to Kentik Detect using such HTTPS sessions, with a Kentik Detect flow ingest server certificate.
  • SNMP is converted into JSON format and NetFlow/sFlow/IPFIX is converted to a Kentik proprietary binary format.

 

 
 top  |  section

Example Proxy Deployment

The following diagram illustrates typical deployment of the Kentik proxy agent.

 

 
 top

chfagent Requirements

The following resources must be available to support the use of chfagent:

  • RAM allocation of 1-2GB per device (varies depending on how “bursty” the flow is).
  • Two CPU cores per 3000 flow records per second (before rate-limiting).

Notes:
- chfagent must be deployed on a separate server from any nProbe host agents sending flow to Kentik Detect via chfagent.
- If chfagent is running on a VM, CPU requirements are higher and CPUs must be dedicated to the instance.
- If the chfagent server will be handling flow ingest of more than 10k FPS then use of a VM is not recommended (may result in dropped flows).

 

 
 top

chfagent Setup

When used for encryption, chfagent pulls information from the Kentik system to determine which routers it will talk to. The routing of flow and SNMP to chfagent is enabled with the following steps:

  1. Create a device in the Kentik Detect portal (Devices » Add Device; see Adding a Device) for each router that you want to send flow from:
    - Set a device’s Name and Description.
    - Set the device type as “Router” (even though you are sending through the agent).
    - Set the Device IP as the IP of the router that the agent will see the flow being received from. You may enter multiple IPs, comma separated, if there is the possibility that flow may source from multiple IPs for the said router. Private IPs are acceptable.
    - Set Device SNMP IP to the router IP that the agent will poll for SNMP.
    - Set Flow Type to the type of flow that the router is configured to export from.
    - Set Sample Rate to the rate at which the router is set to sample.
    - Save (Add) the device.
  2. Download and install chfagent (see chfagent Download and Install).
  3. Check the system clock and timezone settings on the server, which must be accurate to within a minute for chfagent to function correctly. Kentik recommends that hosts running chfagent use Network Time Protocol (NTP).
  4. Determine which of your organization’s users to use for the authentication that enables chfagent to talk to Kentik servers. The designated user may be any user that has been configured in the Users section of the Kentik Detect portal. You may wish to create a user specifically for agent/flow authentication so that this functionality is not tied to a user that is later deactivated (e.g. the person leaves your organization). You’ll need the user’s e-mail address and your organization’s KDE password, available in the User Profile Page.
  5. Determine the IP that chfagent will bind to on your server in order to receive flow.
  6. Determine the port that your server will accept flow on (i.e. where you will point your routers too).
  7. Run chfagent, specifying the arguments described in chfagent Command Line. You’ll likely want to test it initially from the command line and then place it into one of your startup scripts so that it begins on boot. If not placed in the background, the agent will run in the foreground and adhere to standard kill/end signals (e.g. run “nohup chfagent +options &” if running from the command line and exiting shell).
  8. Configure your routers and switches to send flow (see Router Configuration).

 

 
 top

chfagent Download and Install

chfagent is available for both Debian/Ubuntu and CentOS/RHEL. The agent is available for direct download from our downloads page at https://kentik.github.io/. The agent is currently available for the following versions:

Distribution Download path: https://kentik.github.io/downloads/chfagent/...
CentOS/RHEL 5 No longer supported.
CentOS/RHEL 6 rhel/6/chfagent_rhel_6-latest-1.x86_64.rpm
CentOS/RHEL 7 rhel/7/chfagent_rhel_7-latest-1.x86_64.rpm
Debian 7 debian/7/chfagent-wheezy_latest_amd64.deb
Debian 8 debian/8/chfagent-jessie_latest_amd64.deb
Ubuntu 10.04 No longer supported.
Ubuntu 12.04 ubuntu/12.04/chfagent-precise_latest_amd64.deb
Ubuntu 14.04 ubuntu/14.04/chfagent-trusty_latest_amd64.deb
Ubuntu 16.04 ubuntu/16.04/chfagent-xenial_latest_amd64.deb

Note: To upgrade to a current version of chfagent, see Upgrading an Existing chfagent.

The Terminal command for download and install of chfagent varies depending on your Linux variant (download_path is a placeholder; for the actual value see table above):

  • To download and install a Debian/Ubuntu version of the agent, use the following command:

wget https://kentik.github.io/downloads/chfagent/download_path
dpkg -i chfagent*.deb

  • To download and install a CentOS/RHEL version of the agent, use the following command:

wget https://kentik.github.io/downloads/chfagent/download_path
rpm --install chfagent*.rpm

Note: Once chfagent has been downloaded and installed, continue with the steps outlined in chfagent Setup.

 

Upgrading an Existing chfagent

Upgrading an existing instance of chfagent involves the steps shown below.

1. Check the version of the existing chfagent instance.

  • Debian/Ubuntu:

# Command
dpkg -l | grep chfagent
# Response
ii chfagent-latest-ubuntu-16.04 2.3 amd64 no description given

  • Red Hat/Centos/Fedora:

# Command
rpm -qa | grep chfagent
# Response
chfagent-latest-rhel_7-1.0-1.x86_64
# Use that package name in this command:
rpm -qi chfagent-latest-rhel_7-1.0-1.x86_64
# Response
Name : chfagent-latest-rhel_7
Version : 3.3

2. Kill the running chfagent process (all OS versions; placeholders highlighted).

# Command
ps -ef | grep chfagent
# Response
root  9979  9895  0 13:59 pts/0  00:00:00 /usr/bin/chfagent -api_pass password_string -api_email username@domain.suffix
# Command
sudo killall chfagent

3. Remove the existing chfagent package.

  • Debian/Ubuntu

# Command
dpkg -l | grep chfagent
# Response
ii chfagent-latest-ubuntu-16.04 2.3 amd64 no description given
# Command
sudo dpkg -r chfagent-latest-ubuntu-16.04
# Response
(Reading database... 60028 files and directories currently installed.)
Removing chfagent-latest-ubuntu-16.04 (2.3)

  • Red Hat/Centos/Fedora

# Command
rpm -qa | grep chfagent
# Response
chfagent-latest-rhel_7-1.0-1.x86_64
# Command
sudo rpm -e chfagent-latest-rhel_7-1.0-1.x86_64

4. Download and install the latest version of chfagent as described in chfagent Download and Install.

5. Check the version of the newly installed chfagent instance, and compare it with the version from step 1 to confirm successful installation.

6. Restart chfagent with the same command options returned in step 2 in response to the ps command. See step 6 of chfagent Setup.

 

 
 top

chfagent Command Line

The command line arguments used when configuring chfagent as a NetFlow proxy agent are described in the following list.

  • -api_email (required): The email address of a registered user as displayed on that user’s API System page, which is accessed via the API button for that user on the Users page.
  • -api_token (required): A Kentik-generated string that chfagent will use to authenticate a registered user (must be the same user as for -api_email). The API token of a registered user is found on that user’s API System page.
  • -host (required): set to one of the following interface IPs:
    - The IP of a single interface for chfagent to listen on;
    - 0.0.0.0 to listen on all interfaces.
  • -port (optional): Set the port to listen on. If omitted, chfagent defaults to listening on port 9995.

The following example shows the structure of a typical command line using the arguments described above (with placeholder values highlighted):

chfagent -api_email=api_email -api_token=api_token -host=interface_ip

Notes:
- If chfagent fails to launch, add the -verbose flag and try again so that you can provide the output to support@kentik.com in order to facilitate troubleshooting.
- Use -h to return a list of arguments.

 

 
 top

SNMP Configuration File

Note: A local config file should be used to specify SNMP settings only when customer information security policies prohibit the configuration of SNMP settings in the Kentik Detect portal.

By default, the SNMP configuration (SNMP IP and SNMP Community) for a given device that sends flow to Kentik Detect is learned by chfagent from that device’s settings in the Kentik Detect portal (see Device IP & SNMP Settings). There may be circumstances, however, in which it is necessary (e.g. for security compliance) not to specify SNMP settings for a given router in the portal. In this case it is possible instead to specify the settings through chfagent configuration, using the optional -snmp_file command line argument to direct chfagent to get that information from a local config file.

When SNMP is configured with an external file, the required SNMP parameters are set from the values in that file. These values are described in the following table:

Parameter Description
device_id Required: The Kentik assigned ID of the device (router or host).
snmp_comm Required: The device’s SNMP community.
snmp_ip Required: The IP address that should be used to poll the router.
minimize_snmp Optional Boolean:
- if false (default), interface counter will be polled every 10 minutes and interface description every 30 minutes;
- if true, interface counter won’t be polled and interface description will be polled every 6 hours.

The config file stores the required settings as JSON key/value pairs. The following example shows a local SNMP configuration file for two devices, with minimize_snmp set to true for the second device:

{
  "devices": [
    {
      "device_id": 2466,
      "snmp_comm": "device 2466 community string",
      "snmp_ip": "polling.ip.of.first_device"
    },
    {
      "device_id": 2681,
      "snmp_comm": "device 2681 community string",
      "snmp_ip": "polling.ip.of.second_device",
      "minimize_snmp": "true"
    }
  ]
}


 

 
 top

chfagent Debugging

The following tips may be useful in debugging issues related to the use of chfagent:

  • Our article on Router Configuration will guide you through the general setup of routers to work with Kentik Detect.
  • If the chfagent command line argument -metrics was set to stderr then you will receive a checkpoint every minute that indicates how much flow you are receiving from the router. If that count is not increasing then there is an issue between your router and chfagent, either router configuration or chfagent config of communication between them.
  • It may take 2-3 minutes for the agent to download flow templates and begin to process flow. You can expect to receive errors (“ [ipfix_parse_msg] no template for 256, skip data set”) for the first few minutes, after which the errors should stop.
  • Errors will be logged in stdout by default, but if the -syslog flag was used in the chfagent command line then instead they are logged in syslog (see chfagent Command Line for details).