In this article:

Contents Search
   

 

Host Configuration

Note: kprobe (beta) has replaced nProbe as the host agent software used by Kentik Detect to collect traffic data from hosts. While existing nProbe devices will continue to work, it’s no longer possible to create new nProbe devices.

Hosts that send flow data to Kentik Detect do so via kprobe host agent software. The use of kprobe with Kentik Detect is covered in the following topics:

Note: For help installing and configuring kprobe, please contact support@kentik.com.

 

 
 top

About kprobe

Kentik Detect is designed to enable flow monitoring not only of routers and switches but also of hosts, which can send augmented flow data. In addition to flow records (NetFlow, sFlow, IPFIX), this augmented data includes Network Performance Metrics (retransmits, network latency, and application latency) as well as Layer 7 info such as requests/responses for both DNS and HTTP. This information is unified in the Kentik Data Engine (KDE) with other data such as GEO and BGP, providing the user — and our anomaly detection/alerting system — with a comprehensive view of where traffic is originating and terminating, traffic performance relative to Internet routing paths, and actual HTTP and DNS requests.

The host application/agent that enables the above functionality is kprobe, which is agent software that runs on Linux hosts. kprobe enables customers to send encrypted flow records from a host to Kentik Detect. The kprobe agent listens for incoming and outgoing packets on any network interface and generates flow data from the packets received.

Notes:
- kprobe is included with your Kentik Detect subscription or trial.
- Each instance of kprobe will send HTTPS-encrypted data directly to Kentik on port 443. If it’s not possible to connect to the Internet, the data can be sent through an HTTP proxy instead, see Host Flow Via Proxy.

 

 
 top

Host Metrics and Dimensions

Based on the data sent from kprobe, Kentik Detect is able to make available a comprehensive set of host-related metrics and dimensions, which are covered in the following topics:

 

 
 top

kprobe Requirements

The following resources must be available to support the use of kprobe:

  • Up to one CPU core per kprobe instance.
  • RAM allocation of 2GB per instance; actual usage is typically 1GB.

In addition to the above, communication between kprobe and Kentik Detect will require you to enable kprobe to open multiple https sessions to multiple *.kentik.com hosts destined to port 443. Please ensure that any proxies, firewalls, routers, and NAT boxes permit this communication. kprobe will work properly through NAT and proxies.

 

 
 top

kprobe Traffic Capacity

Each monitored interface requires its own individual instance of kprobe, with only one such instance per interface. Each of these instances uses only a single core, which prevents excessive CPU utilization but also determines the volume of traffic that can be handled per interface. The following table provides a very rough guide to how kprobe’s traffic capacity per interface (maximum in-plus-out bits) varies by sampling rate.

Sampling ratio Max traffic volume
1:1 or 1:2 100 Mbps
1:40 500 Mbps
1:80 1 Gbps
1:256 3 Gbps
1:1024 10 Gbps
1:4092 > 10 Gbps

Notes:
- Actual performance is affected by a number of factors. A procedure for matching sample rate to traffic volume is outlined in Setting Sampling Rate.
- If protocol decoding (e.g. DNS/WWW data) is not needed, you can optimize performance by disabling decoding (see Disabling Protocol Decoding).

 

 
 top

Registering kprobe Devices

Each kprobe instance that will be sending flow records to Kentik must be registered as a device with Kentik Detect. Device registration may be handled in either of the following ways:

  • On the Add Device page of the Kentik Detect portal (see Adding a Device).
  • Using the Device Create call from the Device API in V5 Admin APIs.

Registering a host in Kentik Detect involves specifying the fields that are described in the KB topic Device Admin Dialogs. The following information about specific fields will help ensure correct registration of a kprobe host:

  • Device type: Choose type “kprobe-beta.”
  • Device IP(s): Enter the IP address of the host that will be sending data. This can be either public or private. If the device has multiple IPs you can choose any of them.
    Note: This is an identifier used to associate a given device object in Kentik Detect with the set of kprobe instances on a given host. It must be unique within each company’s account.
  • BGP Type: To look at Host Traffic Metrics by path you should assign a BGP table to the device by setting BGP Type to “Use table from another peered device.”

Once a kprobe host is registered as a device it will be represented as a row in the Device List on the Devices page (Admin » Devices; see Device List):

  • The Flow column will show a checkmark to indicate receipt of flow data.
  • The device’s type in the list will be indicated as DNS and its flow type will be indicated as “Kentik” (internal Kentik flow format).
  • Note the value of the new host’s ID. You will need to enter this value in the command line when launching kprobe, thereby linking the newly registered device to a specific kprobe instance.

 

 
 top

kprobe Download and Install

To use kprobe, download the executable to each host that you want to monitor. The executable is available at the following URL:

https://kentik.github.io/

The download is a static binary that is designed run on any recent Linux with no dependencies. After downloading, but before running the kprobe command line, run the following command to set permissions so that the binary is executable (owner read/write/execute, group read/execute, and all read/execute):

# chmod 755 kprobe

Notes:
- kprobe must run as root.
- Before proceeding to configuration of kprobe (see kprobe Command Line) check the system clock and timezone settings on the server, which must be accurate to within a minute for kprobe to function correctly. Kentik recommends that hosts running kprobe use Network Time Protocol (NTP).

 

 
 top

kprobe Configuration

kprobe is configured with command line settings that are covered in the following topics:

 

 
 top  |  section

kprobe Command Line

The command line arguments that configure kprobe as a host agent for use with Kentik Detect are described in the topics below.

 

Standard Setup

The following command line parameters are used for the standard kprobe setup:

  • --email (required): Your email address (assuming that you are a registered Kentik Detect user) as displayed on your User Profile Page, which is accessed by clicking your username at the right of the navbar.
  • --token (required): A Kentik-generated string used to authenticate a registered user (must be the same user as for --email), which is found in the API Token field of the User Profile.
  • --interface (required): The name of the interface that kprobe will monitor, e.g. eth0. Each interface uses its own individual instance of kprobe.
  • One of the following mutually exclusive parameters is also required. Any of these may be used to bind the host to the corresponding device that has been registered in Kentik Detect (see Registering kprobe Devices):
    --device-id (recommended): The Device ID shown for the host in the Device List or returned as the value of the id field in the response JSON of the Device API.
    --device-ip: The IP address (ipV4) shown for the host in the Device List or returned as the value of the sending_ips field in the response JSON of the Device API (see Device JSON).
    --device-if: The name, e.g. eth0, of the interface having the IP Address that the device has been registered to in Kentik Detect. This option may be preferred for deployments involving automated or script based provisioning.
  • --sample (optional; default = 1): The denominator of a ratio that represents captured_flows/total_flows where captured_flows (numerator) always equals one and is not stated. For example, if this parameter is specified as 256 then one out of every 256 flows will be captured. For recommended setting, see Setting Sampling Rate.

The following example shows the structure of a typical command line using the arguments described above (with placeholder values highlighted):

# /usr/local/bin/kprobe --email user@domain.suffix --token user_api_token --interface eth0 --device-id ##### --sample ####

Notes:
- The above example would result in protocol decoding (e.g. DNS/WWW data), which could impact kprobe traffic capacity (see kprobe Traffic Capacity). To disable protocol decoding, use the --no-decode flag (see Optional Features).
- To send encrypted flow from kprobe to Kentik via Kentik’s chfagent (NetFlow proxy agent), use the --proxy-url parameter (Optional Features).

 

Optional Features

The following command line parameters and flags are used to enable optional features and behaviors:

  • --no-decode: This flag disables all protocol decoding (see Disabling Protocol Decoding).
  • --proxy-url: The IP address on which you want chfagent to listen when it is used to forward flow data from kprobe to Kentik (see Host Flow Via Proxy).
    Example: --proxy-url http://proxy.example.com
  • --http-port: Set a port on which to decode HTTP traffic (in addition to port 80). Specify multiple times for multiple additional ports.
    Example: --http-port 8080
  • --promisc: This flag enables promiscuous capture of all network traffic seen by the NIC (see Wikipedia article).

 

Print-related Configuration

The following command line flags are used to print kprobe-related information:

  • -h, --help: Print kprobe CLI usage information.
  • -v: Print verbose output. Specify multiple times to increase verbosity, e.g. -vv.
  • -V, --version: Print kprobe version information.

 

Debug-only Parameters

Consult with Kentik support (support@kentik.com) before using these command line parameters for debugging:

  • --api-url: Kentik API URL. Default value is https://api.kentik.com/api/internal
    Example: --api-url http://example.com/api
  • --flow-url: Kentik flow intake URL. Default value is https://flow.kentik.com/chf
    Example: --flow-url http://example.com/flow
  • --metrics-url: Kentik metrics URL. Default value is https://flow.kentik.com/tsdb
    Example: --metrics-url http://example.com/metrics
  • --snaplen: Optional maximum packet capture length.
    Example: --snaplen 1024

Note: The above parameters and flags should not be changed in normal use.

 

 
 top  |  section

Setting Sampling Rate

General considerations related to setting the flow sampling rate for Kentik Detect devices are covered in Flow Sampling. The following additional examples may help you optimize the sampling rate when using kprobe.

For a host handling 10-20 Gbps:

  • Start with a sample:flow ratio of 1:256 (- -sample 256) or 1:512.
  • Check the FPS reported in the Max FPS 5m column of the Device List (Admin » Devices). The preferred range is 1000-2000. Adjust the sample rate as needed.
  • Check CPU utilization. If you see kprobe at greater than 90% CPU then increase the ratio.

For a host handling only a few hundred Mb/s:

  • Start with a low sample rate such as 10 (- -sample 10).
  • Check FPS in the device list and tune the sample rate to achieve the desired FPS. At this flow volume, about 100 FPS provides good resolution, but you may want to vary that depending on how you are using the data (e.g. lower the sampling value for forensics, or increase it for traffic engineering).

Note: The maximum FPS available for any given device depends on the Plan (see About Plans) to which that device belongs. If the maximum FPS is exceeded, Kentik will downsample.

 

 
 top  |  section

Disabling Protocol Decoding

Collection of DNS/WWW data (see Host Traffic Dimensions) is enabled by default. When monitoring host interfaces where collection of DNS/WWW data is not required, kprobe’s traffic capacity can be optimized by disabling protocol decoding. To disable decoding, add the following optional flag to the kprobe command line.

--no-decode


 

 
 top

Host Flow Via Proxy

In situations where it’s not possible for kprobe to communicate with Kentik directly via the Internet, kprobe can be used in conjunction with an HTTP proxy such as chfagent, Kentik’s NetFlow proxy agent. The proxy agent will enable Kentik customers to route flow data from multiple hosts to Kentik via a single point of contact rather than directly from each individual host. To do so:

 

Configure kprobe for chfagent

To use kprobe with chfagent, add a --proxy-url parameter when configuring kprobe (see kprobe Command Line) and set the value to the IP address on which you want chfagent to listen, as shown in the following example (placeholder in italics):

--proxy-url http://#.#.#.#:2020

 

Configure chfagent for kprobe

The command line arguments used when configuring chfagent for use with kprobe are described in the following list.

  • -api_email (required): The email address of a registered user as displayed on that user’s API System page, which is accessed via the API button for that user on the Users page.
  • -api_token (required): A Kentik-generated string that chfagent will use to authenticate a registered user (must be the same user as for -api_email). The API token of a registered user is found on that user’s API System page.
  • -proxy-http (required): set to the port (e.g. 2020) on which chfagent will listen.

The following example shows the structure of a typical command line using the arguments described above (with placeholder values highlighted):

chfagent -api_email=api_email -api_token=api_token -proxy-http=0.0.0.0:2020

Alternatively, if you prefer to keep the API token invisible, you can hide it in an ENV variable:

KENTIK_API_TOKEN=api_token chfagent -api_email=api_email -proxy-http=0.0.0.0:2020

Notes:
- If chfagent fails to launch, add the -verbose flag and try again so that you can provide the output to support@kentik.com in order to facilitate troubleshooting.
- Use -h to return a list of arguments.