Kentik for GCP

Using Kentik With VPC Flow Logs

Kentik® collects, derives, and correlates a wide variety of network traffic data to create the data store that is used for visualization, monitoring, alerting, and analytics. At its core, this time-series database is built around flow data such as NetFlow, sFlow, and IPFIX. Flow records are collected from two main categories of physical devices in the network infrastructure: routers (including related hardware like switches) and hosts (via a software host agent). We also support the extraction of flow records from VPC flow logs generated by cloud-hosted resources such as VPCs in Google Cloud. For Kentik customers using a hybrid cloud architecture (as shown in the diagram below) network traffic visibility can now extend beyond the on-premises network to encompass the Google Cloud Platform (GCP) as well.

The first step toward including VPC flow records in your Kentik flow data is to contact Kentik Customer Success (support@kentik.com). Once you've jointly determined that it makes sense to send Kentik your VPC flow, the following topics will walk you through the setup process:

Combining GCP and on-prem resources into a hybrid cloud.
 

GCP Process Overview

The handoff of flow from Google VPC to Kentik involves two main phases:

  • You enable VPC flow logs in your account, and set the VPC to export the log to a single “Cloud Pub/Sub topic.”
  • Kentik runs software in its own Google account to consume entries from the Cloud Pub/Sub topic, transform those entries into kflow (our internal protocol for flow records), and export the records to Kentik.

Note: The following additional information is available from Google Cloud VPC Documentation:
- Using VPC Flow Logs
- What Is Cloud Pub/Sub

To enable the above process we'll need to accomplish the following specific tasks, which are covered in greater detail below:

  • In Google Cloud:
    - Enable VPC flow logs for each Google VPC subnet that you'd like to cover with Kentik.
    - Export VPC flow logs to a Cloud Pub/Sub topic.
    - Create a pull subscription to enable the request of entries from the Cloud Pub/Sub topic.
    - Set the permissions that will enable Kentik to access the subscription.
  • In the Kentik portal, create a new Cloud (see About Clouds) pointing to the subscription, which results in the automatic creation of a “cloud device” for each subnet that publishes flow logs to the Pub/Sub topic.

Clouds in the Portal

Successful completion of the tasks listed in the overview above will have the following effect in the Kentik portal:

  • A new Cloud will be shown as an added row in the Kentik portal’s Clouds list (Admin » Clouds). The Cloud will represent the collection of VPC subnets whose logs are pulled from the subscription specified in the Add Cloud dialog.
  • The Devices column in the Clouds list will show one or more cloud devices:
    - Each of these cloud devices will represent one subnet that publishes logs to the Pub/Sub topic to which the Cloud is subscribed.
    - Each flow record ingested into KDE from a given cloud device will include the device’s name in the virtual column i_device_name, enabling you to group-by and filter on the device using the Device Name dimension.

Process Options

Google Cloud users have the following three options for executing the procedures required to set up flow logs and Cloud Sub/Pub:

  • Google Cloud Platform Console: A tool that lets you manage your Google Compute Engine resources through a graphical interface (see Google Cloud Platform Console).
  • Gcloud compute: A command-line tool that enables you to manage Google Compute Engine resources (see gcloud compute).
  • Compute Engine API: A RESTful API that creates and runs virtual machines on Google Cloud Platform (see Compute Engine API).

The steps described in this document assume that you are using Console.

 

GCP Logging Setup Tasks

The tasks required to set up the publishing of VPC Flow Logs to a Pub/Sub topic from which they can be ingested into Kentik are covered in the following topics:

 
top  |  section

Enable VPC Flow Logs

Our first task is to enable VPC flow logs for each VPC subnet that you'd like to cover with Kentik. You have the option of enabling flow logs on both existing and newly created subnets; in this example we’ll go with existing.

To enable flow logs on one of your existing subnets:

  1. In the Google Cloud Platform Console, navigate to your VPC networks page:
    - Click the menu icon (hamburger) at the far left of the main navbar.
    - In the resulting menu, find the Networking section, then choose VPC Network » VPC Networks.
  2. In the table on the resulting page, find the row for the region containing the subnet on which you want to enable flow logs, and click on that subnet (e.g. "default") in the Subnets column.
  3. The resulting Subnet details page will include a list of subnet properties and settings. Click Edit in the toolbar at the top of the page.
  4. Find the Flow logs setting in the list, and set it to On.
  5. Click the Configure Logs button to expand the log configuration settings, then check the Include metadata checkbox, which must be checked for Kentik to ingest flow logs.
  6. If needed set the following additional log configuration settings:
    - Aggregation interval: Kentik will support any selected aggregation interval.
    - Sample rate: By default, Kentik assumes that GCP flow logs will be sent unsampled. If you choose to sample, we'll need to manually configure a multiplier for the corresponding devices. Please email support@kentik.com with your project ID, subscription ID, subnet CIDR, and the configured sample rate.
    Note: The effect of the above configurations on your logging volume, as shown in the Estimated logs generated per day field, gives you a sense of the impact of these settings on your logging costs.
  7. Click Save.

Note: The following additional information is available from Google Cloud VPC Documentation:
- Networks and subnets
- Enabling VPC flow logging

 
top  |  section

Create a New Topic

Next we need to configure Logging, a part of Google's GCP Operations suite that lets you read and write log entries, search and filter logs, export logs, and create logs-based metrics. In this case we need to configure export to create a new Pub/Sub topic. As described by Google, enabling export involves creating a “sink,” which includes:

  • A filter that selects the log entries to export.
  • A destination to which the logs will be exported.

As Logging receives new log entries, they are compared against each sink. If a log entry matches a sink's filter, then a copy of the log entry is written to the destination.

To configure a sink for the export of logs to a Pub/Sub topic:

  1. In Console, navigate to your Logs page:
    - Click the menu icon (hamburger) at the far left of the main navbar.
    - In the resulting menu, find the Operations section, then choose Logging » Logs Router.
  2. At the top of the resulting Logs Router page, choose Create Sink. The resulting page will be a form in which you specify the properties of the new sink. The first step in the form (Sink Details) includes the following fields:
    - Sink name: Enter any name of your choosing (the name need not be Kentik-specific).
    - Sink description: Enter a string that will help others understand what this sink is.
  3. In Sink destination (wizard step 2), make the following settings:
    - Select sink service: Choose Cloud Pub/Sub topic.
    - Select a Cloud Sub/Pub topic: Click Create a Topic.
  4. In the Topic ID field of the resulting popup, enter a name for the topic, then click Create Topic.
  5. In Choose logs to include (wizard step 3), create an inclusion filter to limit logging to flow logs (thereby reducing the cost of logging). To capture all flow logs with this sink, enter the following compound filter in the Build inclusion filter field:
    resource.type="gce_subnetwork" AND log_id("compute.googleapis.com/vpc_flows")
    Note: You may be able to further reduce logging costs with a more restrictive filter that limits logging to a specific port and protocol, subnet, subnet prefix, or VM (see the Google documentation topic Networking queries).
  6. Click the Preview Logs button. A Logs Explorer page will open in a new browser tab. The Query results table will list the logs currently included in the sink.
  7. Back in the Create Sink wizard (on the original browser tab):
    - If the logs listed on the Logs Explorer page aren't the logs that you think the sink should be capturing, check your inclusion filter.
    - If the correct logs are being captured by the sink, click the Next button.
  8. The optional Choose logs to filter out step (wizard step 4) can be skipped unless you want to exclude a specific subset of the logs that you included with your inclusion filter (see step 5 in the Google documentation topic Creating a Sink).
  9. Click the Create Sink button, which completes the wizard and takes you back to the console's Logs Router page. The new sink will appear as a row in the Log Router Sinks list (to edit the sink, choose Edit Sink from the drop-down More menu at the right of the row).

Note: The following additional information is available from Google Cloud VPC Documentation:
- Overview of Logs Export
- Exporting with the Logs Viewer

 
top  |  section

Create a Pull Subscription

Now we need to create a “pull” subscription for Kentik’s flow log collection application. The subscription will enable the application to initiate requests to the Cloud Pub/Sub server so we can retrieve messages from the topic to which you’ll be sending your flow logs.

To create a subscription:

  1. In Console, navigate to your Topics page:
    - Click the menu icon (hamburger) at the far left of the main navbar.
    - In the resulting menu, find the Big Data section, then choose Pub/Sub » Topics.
  2. The left-hand pane of the page will include a list of topics. Find the topic that you created in the previous section, then click on the More icon (vertical dots) at the right of that row (if the icon isn't visible, widen your browser window).
  3. Choose Create Subscription from the topic submenu.
  4. On the resulting Create a subscription page, specify properties for the new subscription:
    - Subscription ID: Enter any string of your choosing (spaces are not valid), which will be appended to the full subscription name, which is shown below the field.
    - Delivery type: Choose “Pull.”
    - Additional settings: Specify the other properties of the subscription (for more information, consult Google's Pull subscription topic at the link below).
  5. Click the Create button. After the subscription is created a popup will confirm success, and you'll then be taken to a Subscription Details page for the new subscription.

Note: The following additional information is available from Google Cloud VPC Documentation:
- Cloud Pub/Sub
- Pull subscription

 
top  |  section

Set Permissions

Now that we have a subscription for the topic we need to permit Kentik to access it from the Google Cloud account on which we receive flow logs. To do this, we add the Kentik account to the subscription as a member:

  1. In Console, navigate to your Subscriptions page:
    - Click the menu icon (hamburger) at the far left of the main navbar.
    - In the resulting menu, find the Big Data section, then choose Pub/Sub » Subscriptions.
  2. The left-hand pane of the page will include a list of Subscription IDs and topic names. Find the topic that you created in the previous section, then click the checkbox at the left of the row. The permissions for that topic will now appear in the Permissions tab in the pane at the right of the page.
  3. Click the Add Member button. The Add members drawer will side out from the right of the page.
  4. Specify a member: In the New members field, enter kentik-vpc-flow@kentik-vpc-flow.iam.gserviceaccount.com.
  5. Specify roles:
    - From the Role drop-down, find "Pub/Sub" in the left-hand column, then choose “Pub/Sub Subscriber.”
    - Click Add Another Role, then repeat, this time choosing “Pub/Sub Viewer.”
  6. Click the Save button. A popup will confirm success. We’ve now completed the Google Cloud portion of the setup.

Note: The following additional information is available from Google Cloud VPC Documentation:
- Permissions and roles

 

Create a Cloud in Kentik

So far we've established a Pub/Sub topic to which flow logs can be published, set one or more VPC subnets to publish to that topic, and enabled Kentik to subscribe to the topic. Assuming that all has gone well, we're now done with setup in GCP. To complete the setup process we’ll move on to the Kentik portal.

The last stage of our workflow is to create a Cloud in Kentik that represents all of the VPC subnets publishing to the topic created above, at which point a “cloud device” will be automatically created in Kentik for each individual subnet.

A new cloud is registered in Kentik via the Setup workflow.

To create a Cloud in the Kentik portal:

  1. Click Settings on the main portal navbar.
  2. At the top of the resulting the Settings page, click the link in the Public Clouds card.
  3. At the top of the resulting Public Clouds page, click the Add GCP Cloud button. You'll be taken to the Monitor Your GCP Cloud page in the Setup section of the portal (see GCP Cloud Setup).
  4. Fill in the following fields under Project Details:
    - Project: The name of the GCP project that contains the Cloud Pub/Sub topic that you created as a destination for the publishing of flow logs from your VPC subnets (see Create a New Topic).
    - Subscription: Enter the name of the subscription that you created to enable Kentik to subscribe to your Pub/Sub topic (see Create a Pull Subscription).
  5. Complete the following settings under Name this Data Source:
    - Name: Enter the name to assign to this cloud data source in Kentik.
    - Description: Enter a description for this data source.
    - Billing Plan: Choose the Kentik plan to which this data source should be assigned (see About Plans).
  6. Click the Save button to save the new cloud and return to the Public Clouds Page.
The Public Clouds page lists the clouds registered with Kentik.

At this point we’ve completed the setup process. On the Settings » Clouds page, you should now be able to see changes to the Clouds list as described in Clouds in the Portal. As time passes and flow records from the VPC are ingested into Kentik you’ll be able to use the names of your cloud devices as group-by and/or filter values for the Device Name dimension in Kentik queries.

© 2014- Kentik

In this article: