Host Configuration

Hosts that send flow data to Kentik can do so via kprobe host agent software, which is covered in the following topics:

• About kprobe
• kprobe Requirements
• kprobe Traffic Capacity
• Registering kprobe Devices
• kprobe Download and Install
• kprobe Configuration
• Host Flow Via Proxy

kprobe is Kentik's Linux host agent for augmented flow monitoring. Beyond standard NetFlow, sFlow, and IPFIX, kprobe adds Network Performance Metrics (retransmits, latency) and Layer 7 details (DNS/HTTP requests/responses). This unified data, combined with GEO and BGP in the Kentik Data Engine, provides a comprehensive view of traffic origin/destination, performance across Internet routing paths, and application-layer activity for both users and for Kentik's anomaly detection/alerting system.

kprobe captures network traffic on Linux hosts and sends encrypted flow records to Kentik over HTTPS (port 443). It's included with your subscription or trial and can use an HTTP proxy if direct internet access isn't available (see Host Flow Via Proxy).

Kentik provides a comprehensive set of host-related metrics and dimensions from kprobe data, detailed in:

• Host Traffic Dimensions: see Host Traffic Dimensions.
• Host Traffic Metrics: see Host Traffic Metrics.

The following resources must be available to support the use of kprobe:

• Up to one CPU core per kprobe instance.
• 2GB RAM allocation per instance; actual usage is typically 1GB.

Communication between kprobe and Kentik requires you to:

• Enable kprobe to open multiple HTTPS sessions to *.kentik.com (or *.kentik.eu for EU clusters) on port 443.
• Ensure proxies, firewalls, routers, and NAT boxes allow this communication. kprobe functions properly through NAT and proxies.

Each interface needs a single kprobe instance using one CPU core. This prevents excessive CPU utilization but also determines the volume of traffic that can be handled per interface. Here's a rough guide to kprobe’s traffic capacity per interface by sampling rate.

Notes:
- Actual performance varies. See Setting Sampling Rate for guidance.
- Disable protocol decoding to optimize performance if not needed (see Disabling Protocol Decoding).

To send flow records to Kentik, each kprobe instance must be registered as a device. Registration can be done via:

• The Add Device page in the Kentik portal (see Add a Device).
• The Device Create call from the Device API in V5 Admin APIs.

Key Registration Fields:

• Device Type: Select “Kentik Host Agent (kprobe).”
• Device IP(s): Enter the host's IP address (public or private). Choose any if multiple IPs are available. Ensure uniqueness within your company's account.
• BGP Type: Set to "Use table from another peered device" to look at Host Traffic Metrics by path.

Note: See more details on these fields inDevice Settings Dialog.

Post-Registration:

• The device appears in the Device List (Settings » Networking Devices) with a checkmark in the Flow column, indicating receipt of flow data.
• The device type is shown as DNS, and the flow type as “Kentik”.
• Note the host’s ID for linking the device to a kprobe instance during launch.

To use kprobe, download and install the executable on each host you wish to monitor:

1. Visit https://packagecloud.io/kentik.
2. Select kprobe from the downloads list.
3. On the kprobe Packages page, ensure a compatible version for your Linux distribution is available.
4. Click Installation in the sidebar.
5. Choose the appropriate package type tab (deb, rpm, node, python, or gem) from the Bash Scripts column.
6. Click Copy to copy the quick install cURL.
7. Execute the cURL in Terminal to download and install the packages.

Note: kprobe requires root access.

kprobe configuration via command line is covered in the following topics:

• kprobe Command Line
• kprobe Optional Features
• Setting Sampling Rate
• Disabling Protocol Decoding
• Enabling OTT DNS Collection

For the standard kprobe setup, use the following arguments:

• --email (required): Your registered Kentik email from your User Profile.
• --token (required): Kentik API token from your User Profile.
• --interface (required): The interface name to monitor, e.g. eth0. (Each interface uses its own instance of kprobe).
• --region (required for EU cluster): Use --region=eu for EU cluster.
• To bind the host to a registered device (see Registering kprobe Devices), use one of these mutually exclusive options:
  --device-id (recommended): The Device ID from the Device List or Device API (see Device JSON).
• --device-ip: The IPv4 address from the Device List or Device API.
• --device-if: The interface name with the registered IP in Kentik (may be preferred for deployments involving automated or script-based provisioning).
• --device-name: The host name from the Device List or Device API (may be preferred for deployments involving automated or script-based provisioning).
  Note: Non-alphanumeric characters in device names are replaced with underscores.
• --device-plan (may be required): The ID of the Kentik plan for a new device (ignored if the device already exists).
• --sample (optional): The denominator for flow sampling ratio (e.g. 256 means one out of every 256 flows is captured). If not specified, the sample rate from the portal's Device General Settings will be used.
  Note: For recommended setting, see Setting Sampling Rate.

Example command line structure with placeholders:

# /usr/local/bin/kprobe --email user@domain.suffix --token user_api_token --interface eth0 --device-id ##### --device-plan ##### --sample ####

Notes:
- Protocol decoding (e.g., DNS/WWW data) is enabled, affecting kprobe Traffic Capacity. Use --no-decode to disable (see kprobe Optional Features).
- Use the optional --proxy-url parameter for encrypted flow via Kentik’s kproxy.
- Use either " " or "=" between argument names and values.

Enable optional features with these command line parameters:

• --no-decode: Disables all protocol decoding (see Disabling Protocol Decoding).
• --proxy-url: Sets the IP for kproxy to forward flow data from kprobe to Kentik (see Host Flow Via Proxy).
  Example: --proxy-url http://proxy.example.com
• --http-port: Decodes HTTP traffic on the specified ports, in addition to port 80.
  Example: --http-port 8080
• --promisc: Enables promiscuous capture of all network traffic seen by the NIC (see Wikipedia article).
• --translate: Replaces IP and port values in kprobe-generated flow records, useful for cloud environments. Format as a comma-separated list in the following order: existing IP, existing port, alternate IP, alternate port.
  Example: --translate 123.45.67.189,80,198.76.54.132,8080
• --device-site: Assigns a new device to the specified site ID (see About Sites).

To start a simple HTTP status server, use these options:

• --status-host: Sets the server's listen address. Defaults to 127.0.0.1 if status-port is set but this parameter is not.
• --status-port: Sets the server's listen port. The server only starts if this is set and non-zero.

Access the server via at http://host:port/v1/status to get basic flow statistics in JSON:

{
  "flows-in": {
    "count": 13889,
    "1m.rate": 76.20784200079075,
    "5m.rate": 40.91552674366451
  },
  "flows-out": {
    "count": 13889,
    "1m.rate": 76.20784200079075,
    "5m.rate": 40.91552674366451
  }
}

Use these options to print kprobe-related information:

• -h, --help: Displays kprobe CLI usage information.
• -v: Outputs verbose information. increase verbosity with multiple uses (e.g., -vv).
• -V, --version: Shows kprobe version information.

Consult Product Support (see Customer Care) before using these options for debugging:

• --api-url: Kentik API URL.
  Default: https://api.kentik.com/api/internal
  Example: --api-url http://example.com/api
• --flow-url: Kentik flow intake URL.
  Default: https://flow.kentik.com/chf
  Example: --flow-url http://example.com/flow
• --metrics-url: Kentik metrics URL.
  Default: https://flow.kentik.com/tsdb
  Example: --metrics-url http://example.com/metrics
• --snaplen: Optional max packet capture length.
  Example: --snaplen 1024

Note: Do not change debug-only parameters in normal use.

For general considerations for setting the flow sampling rate for Kentik devices, see Flow Sampling. For hosts, see Sample Rate for Hosts.

Here are tips for optimizing kprobe sampling rates:

• Hosts handling 10-20 Gbps:
  - Start with a sample:flow ratio of 1:256 (--sample 256) or 1:512.
  - Check the FPS in the Max FPS 5m column of the Device List. Aim for 1000-2000 FPS. Adjust as needed.
  - If kprobe CPU usage exceeds 90%, increase the ratio.

• Hosts Handling a few hundred Mb/s:
  - Start with a sample rate of 10 (--sample 10).
  - Adjust FPS to around 100 for good resolution, varying based on data use (e.g., lower for forensics, higher for traffic engineering).

Note: The maximum FPS depends on the device's Plan (see About Plans). Exceeding it results in downsampling by Kentik.

By default, DNS/WWW data collection is enabled (see Host Traffic Dimensions). To optimize kprobe's traffic capacity when this data isn't needed, disable protocol decoding with the --no-decode option.

When kprobe can't directly communicate with Kentik via the internet, use it with kproxy, Kentik’s NetFlow proxy agent, to route flow data from multiple hosts through a single point. To do so:

• Modify the kprobe configuration as described in Configure kprobe for kproxy.
• Download kproxy as per kproxy Download and Install.
• Configure the kproxy installation as per Configure kproxy for kprobe.

Add --proxy-url with the desired IP for kproxy to listen:

--proxy-url http://#.#.#.#:2020

Use these command line arguments:

• -api_email (required): The registered user's email.
• -api_token (required): The Kentik-generated authenticate token (must be the same user as for -api_email).
• -proxy-http (required): The port for kproxy to listen (e.g., 2020).

Example command:

kproxy -api_email=api_email -api_token=api_token -proxy-http=0.0.0.0:2020

To hide the API token:

KENTIK_API_TOKEN=api_token kproxy -api_email=api_email -proxy-http=0.0.0.0:2020

Notes:
- If kproxy fails to launch, use the -verbose option for detailed output to assist Product Support (see Customer Care) with troubleshooting.
- Use -h to display a list of arguments.