Kentik for IBM Cloud

The following topics cover flow logging in IBM Cloud and the export of flow logs for ingest into Kentik:

Flow logs from a resource in IBM Cloud may be exported to Kentik via the Blueflow agent.
 

About IBM Cloud Flow Logs

The basics of IBM Cloud flow logs are covered in the following topics:

 
top  |  section

IBM Flow Log Overview

IBM Cloud offers an open and secure public cloud, including a hybrid cloud platform, that features advanced data and AI capabilities (see https://www.ibm.com/cloud). Using IBM Cloud’s Flow Logs for VPC feature, you can collect and store flow logs that contain information about the Internet Protocol (IP) traffic to and from network interfaces within IBM Virtual Private Clouds (VPCs).

The collected logs can be ingested into the Kentik Data Engine with Kentik’s Blueflow agent, which processes the logs from an IBM Cloud bucket. The agent converts the logged data to kflow (Kentik’s flow record format), enriches it with other Kentik-collected network data (GeoIP, BGP, etc.), and stores it as flow records in KDE. These records exist alongside flow data from your data center infrastructure and non-IBM cloud resources, so you can see and analyze all of your network traffic data in a single comprehensive environment.

 
top  |  section

IBM Flow Log Structure

IBM Cloud gathers individual flow logs, which each reflect network traffic for a limited duration, into a time-ordered sequence of flow logs that appear in one or more Cloud Object Storage (COS) objects and are written to a user-specified COS bucket. Each COS object is made up of the following parts:

When Kentik ingests the flow logs the fields of flow records in the Kentik Data Engine are populated with data from the corresponding the IBM flow log fields. To minimize the costs associated with log data retention in the cloud, Kentik deletes flow logs after they’ve been ingested into KDE.

 
top  |  section

IBM Flow Log Resources

The following IBM documentation topics provide additional information related to IBM Cloud in general and flow logging in particular:

 

IBM Logging Setup Overview

Configuring the ingest of flow logs from IBM Cloud into Kentik is a multi-stage process involving the following main tasks:

  1. In IBM Cloud, set up flow log generation and collection (see Flow Log Setup in IBM Cloud):
    - If you’re setting up logging for a new IBM Cloud resource, create a COS service instance.
    - Create a storage bucket.
    - Authorize the bucket for log collection.
    - Create a flow log collector to generate logs on the resource and send them to the bucket.
  2. In IBM Cloud, create and specify the following items that are required for flow log export via Kentik’s Blueflow agent (see Export Setup in IBM Cloud):
    - Create an IBM VPC API key.
    - Create and retrieve IBM Cloud Object Storage service credentials.
    - Get the IBM Cloud Service Endpoint for the region containing the bucket with the logs.
  3. In the Kentik portal (see IBM Cloud Setup in Kentik):
    - Create and configure an IBM Cloud data source.
    - Gather Kentik information needed to deploy the Blueflow agent.
  4. Configure and deploy the Kentik Blueflow agent in IBM Cloud as one of the following (see Deploying the Blueflow Agent):
    - a standalone Docker instance running on an IBM Cloud VM; or
    - a Kubernetes deployment in an IBM IKS Cluster.

When configuration is complete, the Blueflow agent will process the logs from the bucket, converting them to kflow (Kentik’s flow record format), enriching them with other Kentik-collected network data (GeoIP, BGP, etc.), and ingesting them into KDE. Once ingested, the logs will be deleted from the IBM Cloud bucket, thereby minimizing cloud storage costs.

 

Flow Log Setup in IBM Cloud

The following topics cover configuration of IBM Cloud to export flow logs to a COS (Cloud Object Storage) bucket:

Note: The flow log setup process is also described in the IBM Cloud documentation topic Creating a flow log collector.

 
top  |  section

Create a COS Service Instance

Collecting flow logs from an IBM Cloud resource involves creating a “COS bucket” in which the logs are collected. Each such bucket exists within a COS Service Instance. If you don’t already have a COS Service Instance within which to create a bucket for your logs, you’ll need to create one (if you already have a bucket to use, you can skip ahead to Create a Bucket):

  1. Log into your IBM Cloud Console.
  2. Enter “Object Storage” into the search field in the main navbar at top.
  3. Click “Object Storage” in the drop-down search results. You’ll be taken to the Create tab of the Cloud Object Storage configuration page.
  4. In the Select a pricing plan pane, click on Standard.
  5. In the Configure your resource pane:
    - Enter a name in the Service Name field.
    - In the Select a resource group drop-down, choose a group. If there’s a group that includes the target VPCs from which you’d like to export flow logs, choose it. If not, choose Default.
    - If desired, enter one or more Resource Tags in the Tags field (tags can be used to help organize your resources).
  6. Click the Create button at the bottom right of the page. When creation of the service instance is complete you’ll be taken to the Cloud Object Storage management page for your new service.
Settings for a new COS service instance that will contain a bucket to collect flow logs.
 
top  |  section

Create a Bucket

The next task in flow logging setup is to create the bucket in which the flow logs for your COS Service Instance will be collected:

  1. Navigate to the landing page for the COS Service Instance in which you want to create the bucket:
    - If you just created the service instance, the Create button in the last step of the previous topic will take you to the correct page.
    - If you’re using a pre-existing service instance, navigate to the page by entering the name of the instance in the search field in the main navbar of the IBM Cloud console, then choosing the instance from the drop-down results list.
  2. On the Overview tab, click in the Create a bucket card, which will take you to the resource list page for the COS instance.
  3. Click in the Custom Bucket card. The bucket configuration settings will appear.
  4. Enter a name for the bucket in the Unique bucket name field.
  5. In the Resiliency section, choose Regional.
  6. From the Location drop-down, chose the same IBM Cloud region as the VPCs from which you will be exporting flow to this bucket.
  7. In the Storage class section, choose Standard.
  8. Click the Create bucket button at the bottom of the bucket creation form. When bucket creation is complete you’ll be taken to the landing page for the bucket.
Creating a new bucket within which to collect flow logs.
 
top  |  section

Authorize Flow Log Creation

Next we’ll create a service authorization, which grants the permissions needed to create flow log objects in our newly created COS bucket.

  1. In the IBM Cloud console’s main navbar, click the drop-down Manage menu and choose Access (IAM).
  2. On the Manage access and users page, click on Authorizations in the left sidebar.
  3. On the Manage authorizations page, click the blue Create button at the top right.
  4. On the Grant a service authorization page, complete the Service Information Fields (see below).
  5. Click Authorize.

Note: If you already have a service authorization in your account with settings as described in Service Information Fields you can skip this step.

Service Information Fields

The following fields must be completed to authorize resources of type “Flow Logs for VPC” to use the COS instance:

  • Source service: Choose VPC Infrastructure Services in Account.
  • Resource type: Choose Flow Logs for VPC.
  • Source resource instance: Choose All resource instances.
  • Target service: Choose Cloud Object Storage in Account.
  • Service instance: Choose string equals for All instances.
  • Service access: Click the checkbox for Writer, which specifies the role via which the source service accesses the target service.
Settings for authorizing the writing of flow logs to a bucket in a COS service instance.
 
top  |  section

Create Flow Log Collector

So far in IBM Cloud we’ve created a COS service instance, a bucket in that instance, and the authorization needed to write logs to the bucket. Now we’ll set up the flow log collector resource that will actually write the logs to the bucket:

  1. In the IBM Cloud console, enter “Flow Logs for VPC” into the search field, then click Flow Logs for VPC in the drop-down results list, which will take you to the New flow log collector page.
  2. In the Name field, enter a unique name for your flow log collector.
  3. In the Resource group drop-down, choose a group. If there’s a group that includes the target VPCs from which you’d like to export flow logs, choose it. If not, choose Default.
  4. If desired, enter one or more tags in the Tags field (tags can be used to help organize your resources).
  5. Use the Attach the flow log connector to radio buttons to choose the type of object from which the flow logs will be generated. You will be asked to specify additional settings depending on the object type.
    - VPC: Log all network traffic within a VPC. Choose the VPC from the Virtual Private Cloud drop-down.
    - Subnet: Log all network traffic within a subnet in a VPC. Choose the VPC from the Virtual Private Cloud drop-down and the subnet from the Subnet drop-down.
    - Instance: Log all network traffic within a VSI in a VPC. Choose the VPC from the Virtual Private Cloud drop-down and the VSI from the Virtual server instance drop-down.
    - Interface: Log all network traffic for an individual interface on a VSI in a VPC. Choose the VPC from the Virtual Private Cloud drop-down, the VSI from the Virtual server instance drop-down, and the interface from the Network Interface drop-down.
  6. Enter the following in the Flow Log Storage pane:
    - Cloud Object Storage instances: Use the drop-down to choose the COS instance that contains the bucket where you want to collect the flow logs.
    - Location: The region the target resource resides in (this setting is not editable).
    - Bucket: Use the drop-down to choose the bucket where you want to collect the flow logs.
  7. In the Summary pane at the right of the New flow log collector page, click the blue Create flow log button.
Creating a flow log collector to write flow logs to a bucket in a COS instance.
 

Export Setup in IBM Cloud

With flow logging itself set up in IBM Cloud, you’ll next need to set up a few additional items on the IBM side that are required to enable export of the logs by Kentik’s Blueflow agent. These tasks are covered in the following topics:

 
top  |  section

IBM VPC API Key

To collect the IBM VPC API key from the IBM Cloud console:

  1. On the main console navbar, click Manage and choose Access (IAM) from the drop-down menu.
  2. On the Manage access and users page, click API keys in the sidebar at left.
  3. On the API keys tab, click the blue Create an IBM Cloud API key button to open the Create API key dialog.
  4. Enter a name and description for the key, then click the Create button.
  5. When the key is generated, click the Download button to download the resulting JSON file to a location from which you can reference it later.
  6. Close the dialog (X at upper right).
 
top  |  section

IBM COS Service Credential

Next, we need to create and retrieve IBM Cloud Object Storage service credentials from the service instance we created in Create a COS Service Instance:

  1. On the main console navbar, enter the name of the service instance into the Search field, then click on the name in the drop-down results list, which will take you to the COS page for the instance.
  2. Click on Service credentials in the sidebar at left.
  3. On the Service credentials tab, click the blue New credential button, which opens the Create credential dialog.
  4. In the Name field, enter a name for the new credential.
  5. Choose Writer from the Role drop-down.
  6. Click the Add button at the lower left of the dialog. The dialog will close and the new credential will appear in the Service credentials list on the COS page when processing is complete.
  7. In the list, click the Copy icon at the right of the row corresponding to the new credential. The credential text will be copied to the clipboard.
  8. Paste the credential into a text file for later reference.

The credential is a JSON object that contains the following two strings, highlighted in the example code below, that will be needed when launching the Blueflow agent:

  • IBM_COS_API_KEY: The apikey field.
  • IBM_SERVICE_INSTANCE: The last segment of the alphanumeric string in the resource_instance_id field, exclusive of the double-colon after and the delimiting colon before.

{
  "apikey": "this_is_a_placeholder_api_key",
  "endpoints": "https://control.cloud-object-storage.cloud.ibm.com/v2/endpoints",
  "iam_apikey_description": "Auto-generated for key ce259489-86dd-4a5e-a60f-91b6ee5b93a2",
  "iam_apikey_name": "PD-test-credentials",
  "iam_role_crn": "crn:v1:bluemix:public:iam::::serviceRole:Writer",
  "iam_serviceid_crn": "crn:v1:bluemix:public:iam-identity::a/f8d5bfabfa31498d97975368be951a20::serviceid:ServiceId-81607ff8-40ee-46b2-8a72-f7df65d101ab",
  "resource_instance_id": "crn:v1:bluemix:public:cloud-object-storage:global:a/f8d5bfabfa31498d97975368be951a20:463b279c-73d0-4554-8f1f-c41f3fd8996f::"
}


 
top  |  section

IBM COS Service Endpoint

The final IBM Cloud element that is required before deploying the Blueflow agent is a Service Endpoint, which will enable the Kentik Blueflow agent to access the COS bucket containing the flow logs. The endpoint, which will vary depending on the region of the bucket (see the IBM Cloud documentation topic Regional Endpoints), is set as follows:

  1. If you’re not still on the COS page for the instance, enter the name of the service instance into the Search field on the main console navbar, then click on the name in the drop-down results list.
  2. From the COS instance page, click Endpoints in the sidebar at left.
  3. Choose Regional from the Select Resiliency drop-down on the Endpoints tab.
  4. From the Select location drop-down, choose the same region that was specified when you created a bucket for logs in Create a Bucket.
    Note: If you’re not sure about the region, you can check on the Buckets tab of the COS instance page (accessed via left sidebar).
  5. In the table showing endpoints, click the Copy icon in the Direct column. The endpoint path (e.g. s3.direct.us-east.cloud-object-storage.appdomain.cloud) will be copied to the clipboard.
  6. Paste the path somewhere where you can refer to it later.
Find the COS service endpoint for the region from which you'll be exporting logs via the Kentik Blueflow agent.
 

IBM Cloud Setup in Kentik

At this point we’ve set up logging for an IBM Cloud resource (e.g. VPC) to a bucket. Now we’ll do the setup on the Kentik side that enables Kentik to export the collected logs from the bucket and ingest them into the Kentik Data Engine. These tasks are covered in the following topics:

 
top  |  section

Create a Cloud in Kentik

Kentik uses the term “data sources” to refer to any source of flow data (see About Data Sources). Data sources are further categorized into “devices” (routers, switches, and other infrastructure hardware) and “clouds,” which each represent a set of resources hosted in a public cloud (AWS, GCP, Azure, or IBM Cloud).

The following steps in the Kentik portal are used to create and configure a data source whose type is cloud:

  1. In Kentik, choose Settings from the main portal navbar.
  2. On the Settings page, click on Public Clouds in the card at the upper right.
  3. On the Public Clouds page, click the Add IBM Cloud button at the upper right.
  4. In the IBM Cloud Cloud dialog, complete the settings described in IBM Cloud Settings (below).
  5. Click the Add IBM Cloud Cloud button to close the dialog and add the new cloud.

IBM Cloud Settings

Complete the following settings to configure a Kentik cloud from an IBM Cloud data source:

  • Name: The name of the cloud. For a cloud from an IBM Cloud, a good name might be the name or ID of the object (VPC, subnet, interface, or instance) for which logs are being written to the bucket.
  • Description: A description of this cloud.
  • Billing plan: The billing plan (see About Plans) to which this cloud should be assigned.
  • Enabled: Switch on to enable the ingest of flow logs for this cloud.
  • Inherit BGP Data from another device: Switch on if you’d like to assign to this cloud the BGP data from an on-premises router.

Note: For questions about any of the above settings, contact support@kentik.com.

 
top  |  section

Gather Kentik Information

In addition to creating a cloud that will represent the logged IBM resource (e.g. VPC) in Kentik, we’ll also need to pull together information from the Kentik portal that is required to set up the Blueflow agent.

To gather the required information from the Kentik portal:

  1. Choose Settings from the main portal navbar.
  2. On the Settings page, click on Public Clouds in the card at the upper right.
  3. In the Clouds list, click the name in the row corresponding to the cloud for which you are setting up logging (e.g. the cloud created in Create a Cloud in Kentik). You’ll navigate to the Cloud page for the cloud.
  4. Beneath the heading at the top of the page you’ll see the Export ID field, which gives the export ID for the cloud; make a note of it.
  5. Choose Licenses from the main portal navbar.
  6. In the Plans pane on the Licenses page, find the billing plan to which you assigned the cloud in the IBM Cloud Cloud dialog (see Create a Cloud in Kentik). The plan ID will appear in parentheses; make a note of it.
  7. Click the User icon at the far right of the main portal navbar, then choose Profile from the drop-down menu, which takes you to the User Information page. Note the address shown in the Email field.
  8. In the sidebar at left, click Authentication, which takes you to the Authentication tab. Note the code shown in the API Token field.
 

Deploying the Blueflow Agent

The final phase of setup for logging IBM Cloud resources is to deploy Kentik’s Blueflow agent, which is covered in the following topics:

 
top  |  section

Blueflow Deployment Overview

Kentik’s Blueflow agent processes the logs from an IBM Cloud bucket, converting them to kflow (Kentik’s flow record format), enriching them with other Kentik-collected network data (GeoIP, BGP, etc.), and ingesting them into KDE. The agent may be installed via a standalone Docker instance running on an IBM Cloud VM or as a Kubernetes deployment directly into your IBM IKS Cluster.

Deployment of the agent directly into your IBM IKS Cluster has the following benefits:

  • Provides HA-reliability and removes a single source of failure.
  • Horizontal scalability for processing large amounts of flow.
  • Redis is used to coordinate from a leader process to followers. The leader will poll for flow and metadata, while the followers will process flow.
 
top  |  section

Environment Variable Values

Whether you deploy via Docker or Kubernetes, you’ll need to provide values for the following environment variables:

  • KENTIK_KEY: Your Kentik API key (see API Token in Gather Kentik Information).
  • IBM_KEY: The IBM VPC API Key that you downloaded as a JSON file.
  • IBM_COS_KEY: The apikey field from the JSON object that you copied to the clipboard in IBM COS Service Credential.
  • SERVICE_INSTANCE: The service instance segment of the resource_instance_id field in the same JSON object.
 
top  |  section

Deploying Blueflow via Docker

The topics below explain how to run Blueflow as a standalone Docker instance on an IBM Cloud VM.

Create an IBM Cloud VM

In IBM Cloud, create a new virtual server in the same region as the bucket created in Create a Bucket:

  1. In the IBM Cloud console, click the menu (hamburger) icon and choose VPC Infrastructure » Virtual server instances from the pop-out sidebar at left.
  2. On the Virtual server instances page, click the blue Create button, which opens the New virtual server for VPC form.
  3. Enter a name in the Name field.
  4. From the Virtual private cloud drop-down, choose the VPC where the VM will be configured to reside, which may be any VPC within the same region that you specified for the bucket in Create a Bucket.
  5. From the Resource Group drop-down, choose the same group you chose in Create a COS Service Instance.
  6. The Location drop-down will show the zones (data centers) that IBM Cloud has determined are available for your new virtual server. Choose the zone in which you wish to deploy the Docker instance.
  7. For the Image setting, choose a version from either the CentOS or Debian GNU/Linux drop-down.
  8. In the Profile pane, click on View all profiles to open the pop-out Select an instance profile form, where you can specify a profile that best fits your needs based on VM utilization.
    Note: For best performance, choose a profile from the Balanced family and modify it if needed.
  9. On the SSH keys drop-down, choose the SSH Key you’ll use to manage this host after the VM is built. If you don’t want to use any of the existing keys, click New key to open the pop-out Add SSH key form.
  10. You need not make changes to the Boot volume, Data volumes, or Network interfaces panes. In the right sidebar, click the blue Create virtual server instance button, which will return you to the Virtual server instances page, where you’ll see the new server instance listed with a status of “Starting.”

Configure the VM for Blueflow

Once you’ve created a VM you’ll need to configure it for Blueflow:

  1. Log into the VM as you would any of your other IBM Cloud VM and obtain root credentials
  2. If needed, install Docker:
    sudo yum install -y yum-utils
    sudo yum-config-manager \
    --add-repo https://download.docker.com/linux/centos/docker-ce.repo
    sudo yum install docker-ce docker-ce-cli
  3. Start the Docker daemon and configure it to restart after a reboot of the VM:
    systemctl start docker
    systemctl enable docker
  4. Using the Linux command line, configure the environment variables listed in Environment Variable Values, which will be used by Docker to protect sensitive data.
  5. Download the Kentik Blueflow agent:
    docker pull kentik/blueflow:v1
  6. Start the Blueflow daemon in your VM by running the command shown in the code snippet below. Before running this code, substitute the correct values for the following fields:
    - bucket_name: The name you specified for the flow log bucket in Create a Bucket.
    - service_endpoint: The path that you copied to the clipboard in IBM COS Service Endpoint.
    - api_email: The email address from your Kentik user profile (see Gather Kentik Information).
    - plan_id and export_id: The ID of the Kentik plan to which the cloud representing this log bucket is assigned, as well as the export ID for that plan (see Gather Kentik Information).

docker run \
--restart unless-stopped \
-d \
--name blueflow \
-p 8083:8083 \
-e IBM_VPC_API_KEY=$IBM_KEY \
-e KENTIK_API_TOKEN=$KENTIK_KEY \
-e IBM_COS_API_KEY=$IBM_COS_KEY \
-e IBM_SERVICE_INSTANCE=$SERVICE_INSTANCE \
kentik/blueflow:v1
\
--bucket_name=my_log_bucket \
--log_level=debug \
--service_endpoint=s3.direct.us-east.cloud-object-storage.appdomain.cloud\
--api_email=kentik_user_email@company.com \
--plan_id=##### \
--export_id=####


 
top  |  section

Deploying Blueflow on IKS

To run Blueflow as a deployment on an IKS cluster:

  1. Configure your workstation to interact with the target cluster.
  2. Clone the Kentik Blueflow configuration repo from GitHub:
    git clone https://github.com/kentik/blueflow-cfg.git
  3. Cd to the resulting directory:
    cd blueflow-cfg/
  4. Using the values listed in Environment Variable Values, use the Linux command line to configure environment variables, which will be used by Kubernetes to protect sensitive data.
  5. In the blueflow-deploy.yaml file, set the following arguments in the args element to the values described below:
    - bucket_name: The name you specified for the flow log bucket in Create a Bucket.
    - service_endpoint: The path that you copied to the clipboard in IBM COS Service Endpoint.
    - api_email: The email address from your Kentik user profile (see Gather Kentik Information).
    - plan_id and export_id: The ID of the Kentik plan to which the cloud representing this log bucket is assigned, as well as the export ID for that plan (see Gather Kentik Information).
  6. Apply the configuration to the Kubernetes cluster:
    kubectl apply -f./blueflow-deploy.yaml
©2014-20 Kentik

In this article: