The following topics cover the use of Kentik’s NetFlow proxy agent, chfagent, to enable encrypted local flow from your organization’s routers and switches to Kentik Detect:
The Kentik NetFlow proxy agent is covered in the following topics:
Kentik allows customers to send the network data collected from their devices to Kentik Detect securely using a downloadable software agent called “chfagent.” Among other functions, chfagent acts as a NetFlow proxy agent that enables the local encryption of flow records (NetFlow v5/v9, IPFIX, and sFlow) before those records are forwarded to Kentik.
The machine running chfagent isn’t actually handling traffic directly, but rather allows flow and SNMP from routers to be locally collected and encrypted for transport to Kentik Detect. A single instance of the chfagent executable can redirect flow for multiple routers and switches, handling rate limiting and resampling as well as encryption. Multiple servers across the network can run chfagent to distribute traffic and load.
The steps involved in downloading, installation, and configuration of chfagent is described in the following topics below:
- The step-by-step routing of flow and SNMP to chfagent is covered in chfagent Setup.
- Downloading chfagent from Kentik is described in chfagent Download and Install.
- Configuration of chfagent for flow encryption is described in chfagent Command Line.
Note: For assistance with any aspect of the setup process, please contact email@example.com.
The diagram below shows an overview of traffic flows between the Kentik NetFlow proxy agent, installed in a customer backbone or IT facility, and Kentik Detect:
The following points describe agent behavior and the connection between customer devices, the Kentik proxy agent, and Kentik Detect:
- chfagent transports all traffic received from customer devices securely to Kentik Detect.
- The Kentik Data Engine and the Kentik Detect portal handle data forwarded via chfagent identically to data received directly from customer devices.
- chfagent connects to any customer devices sending NetFlow telemetry to Kentik Detect, and also to Kentik Detect itself to send the encrypted data as well as to receive configuration information.
- In order to send traffic to Kentik Detect, chfagent will build, for each NetFlow-sending device, two HTTPS sessions, one for flow traffic and the other for SNMP. All traffic is sent to Kentik Detect using such HTTPS sessions, with a Kentik Detect flow ingest server certificate.
- SNMP is converted into JSON format and NetFlow/sFlow/IPFIX is converted to a Kentik proprietary binary format.
The following diagram illustrates typical deployment of the Kentik proxy agent.
The following resources must be available to support the use of chfagent:
- RAM allocation of 1-2GB per device (varies depending on how “bursty” the flow is).
- Two CPU cores per 3000 flow records per second (before rate-limiting).
- chfagent must be deployed on a separate server from any kprobe host agents sending flow to Kentik Detect via chfagent.
- If chfagent is running on a VM, CPU requirements are higher and CPUs must be dedicated to the instance.
- If the chfagent server will be handling flow ingest of more than 10k FPS then use of a VM is not recommended (may result in dropped flows).
When used for encryption, chfagent pulls information from the Kentik system to determine which routers it will talk to. The routing of flow and SNMP to chfagent is enabled with the following steps:
- Create a device in the Kentik Detect portal (Devices » Add Device; see Adding a Device) for each router that you want to send flow from:
- Set a device’s Name and Description.
- Set the device type as “Router” (even though you are sending through the agent).
- Set the Device IP as the IP of the router that the agent will see the flow being received from. You may enter multiple IPs, comma separated, if there is the possibility that flow may source from multiple IPs for the said router. Private IPs are acceptable.
- Set Device SNMP IP to the router IP that the agent will poll for SNMP.
- Set Flow Type to the type of flow that the router is configured to export from.
- Set Sample Rate to the rate at which the router is set to sample.
- Save (Add) the device.
- Download and install chfagent (see chfagent Download and Install).
- Check the system clock and time zone settings on the server, which must be accurate to within a minute for chfagent to function correctly. Kentik recommends that hosts running chfagent use Network Time Protocol (NTP).
- Determine which of your organization’s users to use for the authentication that enables chfagent to talk to Kentik servers. The designated user may be any user that has been configured in the Users section of the Kentik Detect portal. You may wish to create a user specifically for agent/flow authentication so that this functionality is not tied to a user that is later deactivated (e.g. the person leaves your organization). You’ll need the user’s e-mail address and your organization’s KDE password, available in the User Profile.
- Determine the IP that chfagent will bind to on your server in order to receive flow.
- Determine the port that your server will accept flow on (i.e. where you will point your routers too).
- Run chfagent, specifying the arguments described in chfagent Command Line. You’ll likely want to test it initially from the command line and then place it into one of your startup scripts so that it begins on boot. If not placed in the background, the agent will run in the foreground and adhere to standard kill/end signals (e.g. run “nohup chfagent +options &” if running from the command line and exiting shell).
- Configure your routers and switches to send flow (see Router Configuration).
chfagent is available for both Debian/Ubuntu and CentOS/RHEL. The agent is available for direct download from our downloads page at https://kentik.github.io/. The agent is currently available for the following versions:
||Download path: https://kentik.github.io/downloads/chfagent/...
||No longer supported.
||No longer supported.
Note: To upgrade to a current version of chfagent, see Upgrading an Existing chfagent.
The Terminal command for download and install of chfagent varies depending on your Linux variant (download_path is a placeholder; for the actual value see table above):
- To download and install a Debian/Ubuntu version of the agent, use the following command:
dpkg -i chfagent*.deb
- To download and install a CentOS/RHEL version of the agent, use the following command:
rpm --install chfagent*.rpm
Note: Once chfagent has been downloaded and installed, continue with the steps outlined in chfagent Setup.
Upgrading an existing instance of chfagent involves the steps shown below.
1. Check the version of the existing chfagent instance.
dpkg -l | grep chfagent
ii chfagent-latest-ubuntu-16.04 2.3 amd64 no description given
rpm -qa | grep chfagent
# Use that package name in this command:
rpm -qi chfagent-latest-rhel_7-1.0-1.x86_64
Name : chfagent-latest-rhel_7
Version : 3.3
2. Kill the running chfagent process (all OS versions; placeholders highlighted).
ps -ef | grep chfagent
root 9979 9895 0 13:59 pts/0 00:00:00 /usr/bin/chfagent -api_pass password_string -api_email firstname.lastname@example.org
sudo killall chfagent
3. Remove the existing chfagent package.
dpkg -l | grep chfagent
ii chfagent-latest-ubuntu-16.04 2.3 amd64 no description given
sudo dpkg -r chfagent-latest-ubuntu-16.04
(Reading database... 60028 files and directories currently installed.)
Removing chfagent-latest-ubuntu-16.04 (2.3)
rpm -qa | grep chfagent
sudo rpm -e chfagent-latest-rhel_7-1.0-1.x86_64
4. Download and install the latest version of chfagent as described in chfagent Download and Install.
5. Check the version of the newly installed chfagent instance, and compare it with the version from step 1 to confirm successful installation.
6. Restart chfagent with the same command options returned in step 2 in response to the ps command. See step 6 of chfagent Setup.
The setup of chfagent using a CLI involves the following steps:
- Get your API Token:
- In the Kentik Detect portal, click your user name at the right of the main navbar, then choose My Profile from the drop-down menu.
- On the My Profile page, choose Authentication from the sidebar at left.
- On the Authentication page, copy the contents of the field in the API Token pane at bottom.
- Create an environment variable named KENTIK_API_TOKEN whose value is the token you copied from the portal Authentication page.
Note: Once you have an environment variable for the token, do not include the token in the command line with the api_token parameter. Storing the token as an environment variable prevents it from being exposed via a process (ps) command and is consistent with security best practices.
- Set up chfagent using the following command (for parameter definitions, see Command Line Arguments):
chfagent -api_email=api_email -host=interface_ip
The following command line arguments are used when configuring chfagent as a NetFlow proxy agent:
- -api_email (required): The email address of a registered user as displayed on that user’s API System page, which is accessed via the API button for that user on the Users page.
- -host (required): set to one of the following interface IPs:
- The IP of a single interface for chfagent to listen on;
- 0.0.0.0 to listen on all interfaces.
- -port (optional): Set the port to listen on. If omitted, chfagent defaults to listening on port 9995.
- -region (optional): If your organization is registered with Kentik in the EU (rather than the US), set the value of this argument to eu.
If you choose not to store the api_token as an environment variable (step 2 above), despite the risk of exposure via a process (ps) command, your command line will need to include the token using the following additional parameter:
- -api_token: A Kentik-generated string that chfagent will use to authenticate a registered user (must be the same user as for -api_email).
- If chfagent fails to launch, add the -verbose flag and try again so that you can provide the output to email@example.com in order to facilitate troubleshooting.
- Use -h to return a list of arguments.
|Note: A local config file should be used to specify SNMP settings only when customer information security policies prohibit the configuration of SNMP settings in the Kentik Detect portal.
By default, the SNMP configuration (SNMP IP and SNMP Community) for a given device that sends flow to Kentik Detect is learned by chfagent from that device’s settings in the Kentik Detect portal (see Device IP & SNMP Settings). There may be circumstances, however, in which it is necessary (e.g. for security compliance) not to specify SNMP settings for a given router in the portal. In this case it is possible instead to specify the settings through chfagent configuration, using the optional -snmp_file command line argument to direct chfagent to get that information from a local config file.
When SNMP is configured with an external file, the required SNMP parameters are set from the values in that file. These values are described in the following table:
||Required: The Kentik assigned ID of the device (router or host).
||Required: The device’s SNMP community.
||Required: The IP address that should be used to poll the router.
||Required when device type is router:
- If false, interface counter will be polled every 10 minutes and interface description every 30 minutes;
- If true, interface counter won’t be polled and interface description will be polled every 6 hours.
||Optional: Configuration settings required for SNMP v3 (see SNMP V3 Config).
The required settings are stored in the config file as JSON key/value pairs. The following example shows a local configuration file for two devices using SNMP V2, with minimize_snmp set to true for the second device:
"snmp_comm": "device 2466 community string",
"snmp_comm": "device 2681 community string",
When using SNMP v3, the additional required settings are specified with an snmp_v3 object. The object contains the following parameters:
||Required: The user name for SNMP v3 authentication.
||Required: The SNMP v3 authentication protocol:
- NoAuth (must be specified when no authentication is used)
||Required unless AuthenticationProtocol is NoAuth: Password for SNMP V3 authentication.
||Required: The SNMP V3 privacy type:
- NoPriv (must be specified when no privacy is used)
- DES (56-bit encryption)
||Required unless PrivacyProtocol is NoPriv: Password for SNMP V3 privacy.
An example snmp_v3 object is shown in the following sample code:
"snmp_comm": "device 3030 community string",
The following tips may be useful in debugging issues related to the use of chfagent:
- Our article on Router Configuration will guide you through the general setup of routers to work with Kentik Detect.
- The IPs allowed in the Agent tile on the Admin » Access Control page (see Access Control) must include the public IP of the server running chfagent. If the server is behind a NAT gateway you can get its public IP by running wget -qO- ifconfig.co on the server.
- If the chfagent command line argument -metrics was set to stderr then you will receive a checkpoint every minute that indicates how much flow you are receiving from the router. If that count is not increasing then there is an issue between your router and chfagent, either router configuration or chfagent config of communication between them.
- It may take 2-3 minutes for the agent to download flow templates and begin to process flow. You can expect to receive errors (“ [ipfix_parse_msg] no template for 256, skip data set”) for the first few minutes, after which the errors should stop.
- Errors will be logged in stdout by default, but if the -syslog flag was used in the chfagent command line then instead they are logged in syslog (see chfagent Command Line for details).
- A chfagent debug and health-check port is opened by default on the loopback address (127.0.0.1), one port higher than the configured flow ingest port (e.g. for the default flow port of tcp/9995, heath-check is tcp/9996). For further information, contact firstname.lastname@example.org.