Kentik for GCP

Using Kentik to monitor your resources in Google Cloud Platform is covered in the following topics:

Monitoring GCP with Kentik

Kentik^® collects, derives, and correlates a wide variety of network traffic data to create the data store that is used for visualization, monitoring, alerting, and analytics. At its core, this time-series database is built around flow data such as NetFlow, sFlow, and IPFIX. Flow records are collected from two main categories of physical devices in the network infrastructure: routers (including related hardware like switches) and hosts (via a software host agent). We also support the extraction of flow records from VPC flow logs generated by cloud-hosted resources such as VPCs in Google Cloud. For Kentik customers using a hybrid cloud architecture (as shown in the diagram below) network traffic visibility can now extend beyond the on-premises network to encompass the Google Cloud Platform (GCP) as well.

The first step toward including VPC flow records in your Kentik flow data is to contact Kentik Customer Success ([email protected]). Once you've jointly determined that it makes sense to send Kentik your VPC flow, the topics below will walk you through the setup process.

*Combining GCP and on-prem resources into a hybrid cloud.*

GCP Process Overview

The handoff of flow from Google VPC to Kentik involves two main phases:

In you GCP account, you enable flow logs for one or more VPCs in a project, and set the VPC to export the log to a single “Cloud Pub/Sub topic.”
Kentik runs software in its own Google account to consume entries from the Cloud Pub/Sub topic, transform those entries into kflow (our internal protocol for flow records), and export the records to Kentik.

Note:
The following additional information is available from Google Cloud VPC Documentation:
Using VPC Flow Logs
What Is Cloud Pub/Sub

To enable the above process we'll need to accomplish the following specific tasks, which are covered in greater detail below:

In Google Cloud:
- Enable VPC flow logs for each Google VPC subnet that you'd like to cover with Kentik.
- Export VPC flow logs to a Cloud Pub/Sub topic.
- Create a pull subscription to enable the request of entries from the Cloud Pub/Sub topic.
- Set the permissions that will enable Kentik to access the subscription.
In the Kentik portal, create a new cloud export (see Cloud Overview) pointing to the subscription, which results in the automatic creation of a “cloud device” for each subnet that publishes flow logs to the Pub/Sub topic.

Cloud Exports in the Portal

Successful completion of the tasks listed in the overview above will have the following effect in the Kentik portal:

A new cloud export will be added to the Kentik portal’s Cloud Exports list (Settings » Public Clouds). The cloud export will represent the collection of VPC subnets whose logs are pulled from the subscription specified on the Monitor your GCP Cloud page (accessed via the Add GCP Cloud button on the Public Clouds page).
The Devices column in the Cloud Exports list will indicate one or more cloud devices:
- Each of these cloud devices will represent one subnet that publishes logs to the Pub/Sub topic to which the Cloud is subscribed.
- Each flow record ingested into KDE from a given cloud device will include the device’s name in the virtual column i_device_name, enabling you to group-by and filter on the device using the Device Name dimension.

Process Options

Google Cloud users have the following three options for executing the procedures required to set up flow logs and Cloud Sub/Pub:

Google Cloud Platform Console: A tool that lets you manage your Google Compute Engine resources through a graphical interface (see Google Cloud Platform Console).
Gcloud compute: A command-line tool that enables you to manage Google Compute Engine resources (see gcloud compute).
Compute Engine API: A RESTful API that creates and runs virtual machines on Google Cloud Platform (see Compute Engine API).

The steps described in this document assume that you are using Console.

GCP Logging Setup Tasks

The tasks required to set up the publishing of VPC Flow Logs to a Pub/Sub topic from which they can be ingested into Kentik are covered in the following topics:

Enable VPC Flow Logs

Our first task is to enable VPC flow logs for each VPC subnet that you'd like to cover with Kentik, which you'll do in the GCP project containing those VPCs. You have the option of enabling flow logs on both existing and newly created subnets; in this example we’ll go with existing.

To enable flow logs on one of your existing subnets:

In the console, the field just to the right of "Google Cloud" shows the current project. If that's not the project containing the VPCs you'd like to monitor, click in the field to open a dialog in which you can choose a different project.
Navigate to the project's VPC networks page:
1. Click the menu icon (hamburger) at the far left of the main navbar.
2. In the resulting menu, find the Networking section, then choose VPC Network » VPC Networks.
In the table on the resulting page, find the row for the region containing the subnet on which you want to enable flow logs, and click on that subnet (e.g. "default") in the Subnets column.
The resulting Subnet details page will include a list of subnet properties and settings. Click Edit in the toolbar at the top of the page.
Find the Flow logs setting in the list, and set it to On.
Click the Configure Logs button to expand the log configuration settings, then check the Include metadata checkbox, which must be checked for Kentik to ingest flow logs.
If needed set the following additional log configuration settings:
1. Aggregation interval: Kentik will support any selected aggregation interval.
2. Sample rate: By default, the log entry volume is scaled by 0.5 (50%), which means that half the log entries are kept. This can be set from 1.0 (100%) to 0.0 (0%), where a sample rate of 1.0 retains 100% of the logs.
  Note:
  The effect of the above configurations on your logging volume, as shown in the Estimated logs generated per day field, gives you a sense of the impact of these settings on your logging costs.
Click Save.

Note:
The following additional information is available from Google Cloud VPC Documentation:
Networks and subnets
Enabling VPC flow logging

Create a New Topic

Next we need to configure Logging, a part of Google's GCP Operations suite that lets you read and write log entries, search and filter logs, export logs, and create logs-based metrics. In this case we need to configure export to create a new Pub/Sub topic. As described by Google, enabling export involves creating a “sink,” which includes:

A filter that selects the log entries to export.
A destination to which the logs will be exported.

As Logging receives new log entries, they are compared against each sink. If a log entry matches a sink's filter, then a copy of the log entry is written to the destination.

To configure a sink for the export of logs to a Pub/Sub topic:

In Console, check that you're in the same project in which you initiated logging (see Enable VPC Flow Logs), then navigate to your Logs page:
1. Click the menu icon (hamburger) at the far left of the main navbar.
2. In the resulting menu, find the Operations section, then choose Logging » Logs Router.
At the top of the resulting Logs Router page, choose Create Sink. The resulting page will be a form in which you specify the properties of the new sink. The first step in the form (Sink Details) includes the following fields:
1. Sink name: Enter any name of your choosing (the name need not be Kentik-specific).
2. Sink description: Enter a string that will help others understand what this sink is.
In Sink destination (wizard step 2), make the following settings:
1. Select sink service: Choose Cloud Pub/Sub topic.
2. Select a Cloud Sub/Pub topic: Click Create a Topic.
In the Topic ID field of the resulting popup, enter a name for the topic, then click Create Topic.
In Choose logs to include (wizard step 3), create an inclusion filter to limit logging to flow logs (thereby reducing the cost of logging). To capture all flow logs with this sink, enter the following compound filter in the Build inclusion filter field:
resource.type="gce_subnetwork" AND log_id("compute.googleapis.com/vpc_flows")
Note:
You may be able to further reduce logging costs with a more restrictive filter that limits logging to a specific port and protocol, subnet, subnet prefix, or VM (see the Google documentation topic Networking queries).
Click the Preview Logs button. A Logs Explorer page will open in a new browser tab. The Query results table will list the logs currently included in the sink.
Back in the Create Sink wizard (on the original browser tab):
1. If the logs listed on the Logs Explorer page aren't the logs that you think the sink should be capturing, check your inclusion filter.
2. If the correct logs are being captured by the sink, click the Next button.
The optional Choose logs to filter out step (wizard step 4) can be skipped unless you want to exclude a specific subset of the logs that you included with your inclusion filter (see step 5 in the Google documentation topic Creating a Sink).
Click the Create Sink button, which completes the wizard and takes you back to the console's Logs Router page. The new sink will appear as a row in the Log Router Sinks list (to edit the sink, choose Edit Sink from the drop-down More menu at the right of the row).

Note:
The following additional information is available from Google Cloud VPC Documentation:
Overview of Logs Export
Exporting with the Logs Viewer

Create a Pull Subscription

Now we need to create a “pull” subscription for Kentik’s flow log collection application. The subscription will enable the application to initiate requests to the Cloud Pub/Sub server so we can retrieve messages from the topic to which you’ll be sending your flow logs.

In Console, check that you're in the same project in which you created a topic for logging (see Create a New Topic), then navigate to your Topics page:
1. Click the menu icon (hamburger) at the far left of the main navbar.
2. In the resulting menu, find the Big Data section, then choose Pub/Sub » Topics.
The left-hand pane of the page will include a list of topics. Find the topic that you created in the previous section, then click on the More icon (vertical dots) at the right of that row (if the icon isn't visible, widen your browser window).
Choose Create Subscription from the topic submenu.
On the resulting Create a subscription page, specify properties for the new subscription:
1. Subscription ID: Enter any string of your choosing (spaces are not valid), which will be appended to the full subscription name, which is shown below the field.
2. Delivery type: Choose “Pull.”
3. Additional settings: Specify the other properties of the subscription (for more information, consult Google's Pull subscription topic at the link below).
Click the Create button. After the subscription is created a popup will confirm success, and you'll then be taken to a Subscription Details page for the new subscription.

Note:
The following additional information is available from Google Cloud VPC Documentation:
Cloud Pub/Sub
Pull subscription

Set Permissions

Now that we have a subscription for the topic we need to permit Kentik to access it from the Google Cloud account on which we receive flow logs. To do this, we add the Kentik account to the subscription as a member:

In Console, check that you're in the same project in which you created a subscription for logging (see Create a Pull Subscription), then navigate to your Subscriptions page:
1. Click the menu icon (hamburger) at the far left of the main navbar.
2. In the resulting menu, find the Big Data section, then choose Pub/Sub » Subscriptions.
The left-hand pane of the page will include a list of Subscription IDs and topic names. Find the topic that you created in the previous section, then click the checkbox at the left of the row. The permissions for that topic will now appear in the Permissions tab in the pane at the right of the page.
Click the Add Member button. The Add members drawer will side out from the right of the page.
Specify a member: In the New members field, enter [email protected].
Specify roles:
1. From the Role drop-down, find "Pub/Sub" in the left-hand column, then choose “Pub/Sub Subscriber.”
2. Click Add Another Role, then repeat, this time choosing “Pub/Sub Viewer.”
Click the Save button. A popup will confirm success. We’ve now completed the Google Cloud portion of the setup.

Notes:
Depending on your organization's settings, when granting permissions in GCP you may be required to provide Kentik's DIRECTORY_CUSTOMER_ID: C02mivkib.
The following additional information is available from Google Cloud VPC Documentation:
Permissions and roles

Grant Metadata Access

The Kentik Map module of the Kentik portal provides a graphical representation of your Hybrid IT network infrastructure. To display the topology of your GCP resources in the map, Kentik needs to fetch metadata (via the Google Compute Engine API) for each cloud export that you create in the Kentik Portal (see Create a Kentik Cloud Export). To enable us to do so, you will need to grant metadata access permissions to Kentik for the [email protected] service account that you specified above in Set Permissions.

The easiest way to grant this access is to:

Add the [email protected] service account as a Principal to each GCP Project from which you plan to create a cloud export in Kentik;
Grant to the service account the role of Compute Viewer, which will allow Kentik to get and list GCE resources (read-only access, with no ability to read stored data).

To grant the required metadata access:

In Console, check that you're in the same project in which you set up logging (see GCP Logging Setup Tasks), then navigate to the IAM page:
1. Click the hamburger icon at upper left, which shows the sidebar menu.
2. Choose IAM & Admin » IAM to open the IAM page (which defaults to the Permissions tab).
Click the Grant Access button toward the upper left of the page. The Grant Access drawer will open from the right.
In the Add Principals field, paste in the service account:
[email protected]
Click in the Assign Roles field to open a drop-down listing roles, then enter "Compute Viewer" into the filter field.
In the resulting list, click on Compute Viewer (not Compute Network Viewer). The drop-down will close, leaving "Compute Viewer" as the value of the field.
Click the Save button to close the drawer and return to the Permissions tab, where you'll see a table.
In the View by Principals tab of the table, find the service account in the Principal column, and confirm that Compute Viewer is listed in the Role column.

Notes:
To grant permissions as described in the above procedure you'll need the following permissions for the Google project: compute.networks.list and resourcemanager.projects.setIamPolicy.
If you have multiple Google Projects nested in a folder structure, you could optionally add the service account [email protected] as a Principal to the top-level folder and assign to it the “Compute Viewer” role, enabling each nested project to inherit the needed permissions.

Create a Kentik Cloud Export

So far we've established a Pub/Sub topic to which flow logs can be published, set one or more VPC subnets to publish to that topic, and enabled Kentik to subscribe to the topic. Assuming that all has gone well, we're now done with setup in GCP. To complete the setup process we’ll move on to the Kentik portal.

The last stage of our workflow is to create a "cloud export" in Kentik (see Cloud Exports and Devices) that represents all of the VPC subnets publishing to the topic created above, at which point a “cloud device” will be automatically created in Kentik for each individual subnet.

*A new cloud is registered in Kentik via the Setup workflow.*

To create a cloud export in the Kentik portal:

Click Settings on the main portal navbar.
At the top of the resulting Settings page, click the link in the Public Clouds card.
At the top of the resulting Public Clouds page, click the Add GCP Cloud button. You'll be taken to the Monitor Your GCP Cloud page in the Setup section of the portal (see GCP Cloud Setup).
Fill in the following fields under Project Details:
1. Project: The ID of the GCP project that contains the Cloud Pub/Sub topic that you created as a destination for the publishing of flow logs from your VPC subnets (see Create a New Topic).
2. Subscription: Enter the name of the subscription that you created to enable Kentik to subscribe to your Pub/Sub topic (see Create a Pull Subscription).
Complete the following settings under Name this Data Source:
1. Name: Enter the name to assign to this cloud data source in Kentik.
2. Description: Enter a description for this data source.
3. Billing Plan: Choose the Kentik plan to which this data source should be assigned (see About Plans).
Click the Save button to save the new cloud and return to the Public Clouds Page.

*The Public Clouds page lists the clouds registered with Kentik.*

At this point we’ve completed the setup process:

On the Settings » Public Clouds page, you should now be able to see changes to the Cloud Exports list as described in Cloud Exports in the Portal.
You can check whether your cloud export is correctly configured by looking at the GCP row in the Cloud Config Status pane of the portal's Public Clouds page. If an error is shown, click View Details to go to the GCP Configuration Status page. Then click the table row showing the error to open the Details drawer, where you'll be able to hover over the error icon to open a popup explaining the error.

*Error reported for a cloud export that's missing permissions for metadata access.*

As time passes and flow records from the VPC are ingested into Kentik you’ll be able to use the names of your cloud devices as group-by and/or filter values for the Device Name dimension in Kentik queries, and you’ll be able to see the topology of your GCP cloud export in the Kentik Map.