CH logo® Knowledge Base
Contents Search
 
 

 

Kentik for GCP

Using Kentik Detect With VPC Flow Logs

Kentik Detect® collects, derives, and correlates a wide variety of network traffic data to create the data store that is used for visualization, monitoring, alerting, and analytics. At its core, this time-series database is built around flow data such as NetFlow, sFlow, and IPFIX. Flow records are collected from two main categories of physical devices in the network infrastructure: routers (including related hardware like switches) and hosts (via a software host agent). We also support the extraction of flow records from VPC flow logs generated by cloud-hosted resources such as VPCs in Google Cloud. For Kentik customers using a hybrid cloud architecture (as shown in the diagram below) network traffic visibility can now extend beyond the on-premises network to encompass the Google Cloud Platform (GCP) as well.

The first step toward including VPC flow records in your Kentik flow data is to contact Kentik Customer Success (support@kentik.com). Once you’ve jointly determined that it makes sense to send Kentik your VPC flow, the following topics will walk you through the setup process:

 

 
 top

GCP Process Overview

The handoff of flow from Google VPC to Kentik Detect involves two main phases:

  • You enable VPC flow logs in your account, and set the VPC to export the log to a single “Cloud Pub/Sub topic.”
  • Kentik runs software in its own Google account to consume entries from the Cloud Pub/Sub topic, transform those entries into kflow (our internal protocol for flow records), and export the records to Kentik Detect.

Note: The following additional information is available from Google Cloud VPC Documentation:
- Using VPC Flow Logs
- What Is Cloud Pub/Sub

To enable the above process we’ll need to accomplish the following specific tasks, which are covered in greater detail below:

  • In Google Cloud:
    - Enable VPC flow logs for each Google VPC subnet that you’d like to cover with Kentik Detect.
    - Export VPC flow logs to a Cloud Pub/Sub topic.
    - Create a pull subscription to enable the request of entries from the Cloud Pub/Sub topic.
    - Set the permissions that will enable Kentik to access the subscription.
  • In the Kentik Detect portal, create a new Cloud (see About Clouds) pointing to the subscription, which results in the automatic creation of a “cloud device” for each subnet that publishes flow logs to the Pub/Sub topic.

 

Clouds in the Portal

Successful completion of the tasks listed in the overview above will have the following effect in the Kentik Detect portal:

  • A new Cloud will be shown as an added row in the Kentik portal’s Clouds list (Admin » Clouds). The Cloud will represent the collection of VPC subnets whose logs are pulled from the subscription specified in the Add Cloud dialog.
  • The Devices column in the Clouds list will show one or more cloud devices:
    - Each of these cloud devices will represent one subnet that publishes logs to the Pub/Sub topic to which the Cloud is subscribed.
    - Each flow record ingested into KDE from a given cloud device will include the device’s name in the virtual column i_device_name, enabling you to group-by and filter on the device using the Device Name dimension.

 

Process Options

Google Cloud users have the following three options for executing the procedures required to set up flow logs and Cloud Sub/Pub:

  • Google Cloud Platform Console: A tool that lets you manage your Google Compute Engine resources through a graphical interface (see Google Cloud Platform Console).
  • Gcloud compute: A command-line tool that enables you to manage Google Compute Engine resources (see gcloud compute).
  • Compute Engine API: A RESTful API that creates and runs virtual machines on Google Cloud Platform (see Compute Engine API).

The steps described in this document assume that you are using Console.

 

 
 top

GCP Logging Setup Tasks

The tasks required to set up the publishing of VPC Flow Logs to a Pub/Sub topic from which they can be ingested into Kentik Detect are covered in the following topics:

 

 
 top  |  section

Enable VPC Flow Logs

Our first task is to enable VPC flow logs for each VPC subnet that you’d like to cover with Kentik Detect. You have the option of enabling flow logs on both existing and newly created subnets; in this example we’ll go with existing.

To enable flow logs on one of your existing subnets:

  1. In the Google Cloud Platform Console, navigate to your VPC networks page:
    - Click the menu icon (hamburger) at the far left of the main navbar.
    - In the resulting menu, find the Networking section, then choose VPC Network » VPC Networks.
  2. In the Subnets column, click the subnet on which you want to enable flow logs.
  3. The resulting Subnet details page will include a list of subnet properties and settings. Click Edit in the toolbar at the top of the page.
  4. Find the Flow logs setting in the list, and set it to On.
  5. Click Save.

Note: The following additional information is available from Google Cloud VPC Documentation:
- Networks and subnets
- Enabling VPC flow logging

 

 
 top  |  section

Create a New Topic

Next we need to configure Stackdriver Logging, a part of Google’s GPC Stackdriver suite that lets you read and write log entries, search and filter logs, export logs, and create logs-based metrics. In this case we need to configure export to create a new Pub/Sub topic. As described by Google, enabling export involves creating a “sink,” which includes:

  • A filter that selects the log entries to export.
  • A destination to which the logs will be exported.

As Stackdriver Logging receives new log entries, they are compared against each sink. If a log entry matches a sink’s filter, then a copy of the log entry is written to the destination.

To configure a sink for the export of logs to a Pub/Sub topic:

  1. In Console, navigate to your Logs page:
    - Click the menu icon (hamburger) at the far left of the main navbar.
    - In the resulting menu, find the Stackdriver section, then choose Logging » Logs.
  2. On the resulting logs viewer page, choose “GCE Subnetwork” from the left-most drop-down menu (under the filter field).
  3. On the next menu to the right (which defaults to “All Logs”), choose “vpc_flows” and click OK.
  4. In the toolbar at the top of the page, click Create Export.
  5. In the Edit Export sidebar that pops over the right of the page (see screenshot), specify the properties of the new sink:
    - Sink Name: Enter any name of your choosing (the name need not be Kentik-specific).
    - Sink Service: Choose Cloud Pub/Sub.
    - Sink Destination: Choose “Create new Cloud Pub/Sub topic.” In the resulting modal, enter any topic name of your choosing.
    Note: Invalid characters, e.g. spaces, will prevent activation of the Create Sink button (no error message).
  6. Click the Create Sink button. After the sink is saved a modal will open to confirm success and provide information about the new sink.

Note: The following additional information is available from Google Cloud VPC Documentation:
- Overview of Logs Export
- Exporting with the Logs Viewer

 

 
 top  |  section

Create a Pull Subscription

Now we need to create a “pull” subscription for Kentik’s flow log collection application. The subscription will enable the application to initiate requests to the Cloud Pub/Sub server so we can retrieve messages from the topic to which you’ll be sending your flow logs.

To create a subscription:

  1. In Console, navigate to your Topics page:
    - Click the menu icon (hamburger) at the far left of the main navbar.
    - In the resulting menu, find the Big Data section, then choose Pub/Sub » Topics.
  2. The left-hand pane of the page will include a list of topics. Find the topic that you created in the previous section, then click on the submenu icon at the right of that row (if the icon isn’t visible, widen your browser window).
  3. Choose New Subscription from the topic submenu.
  4. In the resulting Create a subscription page, specify properties for the new subscription:
    - Subscription name: Enter any name of your choosing (spaces are not valid).
    - Delivery type: Choose “Pull”
  5. Click the Create button. After the subscription is created a popup will confirm success.

Note: The following additional information is available from Google Cloud VPC Documentation:
- Cloud Pub/Sub
- Pull subscription

 

 
 top  |  section

Set Permissions

Now that we have a subscription for the topic we need to permit Kentik to access it from the Google Cloud account on which we receive flow logs. To do this, we add the Kentik account to the subscription as a member:

  1. In Console, navigate to your Subscriptions page:
    - Click the menu icon (hamburger) at the far left of the main navbar.
    - In the resulting menu, find the Big Data section, then choose Pub/Sub » Subscriptions.
  2. The left-hand pane of the page will include a list of topics. Find the topic that you created in the previous section, then click the checkbox at the left of the row. The permissions for that topic will now appear in the Permissions pane at the right of the page.
  3. Specify properties for a new member:
    - Add members: Enter “kentik-vpc-flow@kentik-vpc-flow.iam.gserviceaccount.com”
    - Select a role: Choose both “Pub/Sub Subscriber” and “Pub/Sub Viewer”
  4. Click the Add button. A popup will confirm success. We’ve now completed the Google Cloud portion of the setup.

Note: The following additional information is available from Google Cloud VPC Documentation:
- Permissions and roles

 

 
 top

Create a Cloud in Kentik

So far we’ve established a Pub/Sub topic to which flow logs can be published, set one or more VPC subnets to publish to that topic, and enabled Kentik to subscribe to the topic. Assuming that all has gone well, we’re now done with setup in GCP. To complete the setup process we’ll move on to the Kentik Detect portal.

The last stage of our workflow is to create a Cloud in Kentik Detect that represents all of the VPC subnets publishing to the topic created above, at which point a “cloud device” will be automatically created in Kentik Detect for each individual subnet.

To create a Cloud in the Kentik Detect portal:

  1. Click Admin on the main portal navbar.
  2. At the top of the resulting the Admin page you’ll see the Add Data Sources pane. In the Google Cloud Platform tile, click the Add button to open the Add GCP Cloud dialog (see Cloud Admin Dialogs).
  3. Enter the Name and Description for the Cloud.
  4. Assign the Cloud to a billing plan (see About Plans).
  5. Set the Enabled switch to On.
  6. Enter the GCP Provider Settings:
    - Project: The name of the GCP project that contains the Cloud Pub/Sub topic that you created as a destination for the publishing of flow logs from your VPC subnets (see Create a New Topic).
    - Subscription: Enter the name of the subscription that you created to enable Kentik to subscribe to your Pub/Sub topic (see Create a Pull Subscription).
  7. Click the Add GCP Cloud button to save the new Cloud and return to the Clouds Page.

At this point we’ve completed the setup process. On the Admin » Clouds page, you should now be able to see the Clouds list changes described in Clouds in the Portal. As time passes and flow records from the VPC are ingested into Kentik Detect you’ll be able to use the names of your cloud devices as group-by and/or filter values for the Device Name dimension in Kentik Detect queries.

 

In this article: