Cloud Setup

The setup of cloud instances is covered in the following topics:

• About Cloud Setup
• AWS Cloud Setup
• Azure Cloud Setup
• GCP Cloud Setup
• OCI Cloud Setup

Cloud setup is one setup task within the overall onboarding process that new Kentik customers (or trial users) go through to get up and running with Kentik (see Setup Overview). The result of a cloud setup procedure is to create one cloud export (listed on the Public Clouds Page at Settings » Public Clouds) that has one or more cloud devices (listed on the details page for an individual cloud export) representing the resources that are monitored via the cloud export. You can have multiple cloud exports from a given cloud provider.

The cloud-related tasks listed at the left of the Welcome to Network Observability page depend on which cloud providers you check in the Which Data page (see Initial Setup Login). Each cloud setup process involves working through a wizard that gathers needed information. The workflow varies depending on the cloud provider. The steps for each Kentik-supported cloud provider are listed in the topics later in this article.

Note: The setup pages used to add a cloud export during your organization's initial Kentik onboarding are also used to add cloud export at any later time via the Settings » Public Clouds page.

To begin a cloud setup task:

1. In the Setup Tasks list, click the Monitor item for a given cloud provider.
2. In the main display area to the right of the list, click the Get Started button, which takes you to the Monitor page for your selected cloud provider (e.g. the Monitor Your AWS Cloud page).

Note: When working through the wizard the information you enter is saved only pages with a Save button. On all other pages, if you use the Back button to go upstream in the wizard the information you've entered on the page will be lost.

The process of setting up to monitor a cloud resource in AWS is covered in the following topics:

• About AWS Cloud Setup
• AWS Manual Setup
• AWS Automated Setup

Kentik supports ingest of flow logs from one or more Amazon Web Services cloud instance (VPC, subnet, or network interface). Collected in an AWS S3 bucket, the logs are brought into Kentik via a "cloud export" that is configured on the Kentik portal's Monitor your AWS Cloud page and managed via the Public Clouds Page (Settings » Public Clouds). By default, each VPC, subnet, or interface sending logs to the bucket will be represented in Kentik as a "cloud device."

Note: For more information on AWS flow log formats, log deletion, and AWS flow log documentation: About AWS Cloud Visibility.

The initial setup of an AWS cloud in Kentik can be handled in either of the following ways:

• Manual Configuration: Configure without automated assistance; see AWS Manual Setup.
  - In the AWS console, you'll configure log generation and collection manually.
  - In Kentik, you'll enter the information needed to register a cloud export that will bring the logs from AWS, normalize them for Kentik, and ingest them into the Kentik Data Engine.
• Automated Configuration: You'll enter information for both the AWS configuration and the Kentik cloud export into a single form in Kentik. The configuration in AWS will be performed by Terraform (see AWS Automated Setup).

Upon arrival at the Monitor your AWS Cloud page (see Starting Cloud Setup above), proceed with configuring a cloud export by choosing the tab corresponding to the type of setup you prefer (manual or automated).

The Manual Configuration tab is used for manual setup of a cloud export. Unlike the Automated Configuration tab, this tab assumes that the configuration of flow logging in AWS is already complete. The information you enter is used to register a cloud export, which brings the logs into Kentik from AWS.

Kentik supports multiple "cloud exports" from AWS, each of which is specific to one combination of AWS region and role. A cloud export enables us to access two kinds of data from your AWS resources (VPCs, subnets, or network interfaces):

• Flow logs (all versions) for traffic analytics (Network Explorer, Insights, Alerting, etc.);
• Metadata for topology views on the Kentik Map, including regions, availability zones, VPC IDs, etc.

AWS allows flow logs from resources in one region to be collected in a "centralized" bucket in a different region, but metadata must be collected directly from each region. To include all of your AWS regions in the AWS topology views of the Kentik map you would create the following types of cloud exports:

• Flow + metadata: Used for each region that has a bucket for collecting flow logs.
• Metadata-only: Used for each region whose logs are sent to a centralized bucket in a different region.

On the Monitor your AWS Cloud page:

1. Select the Manual Configuration tab.
2. In the radio buttons toward the top of the form, choose the type of cloud export (see AWS Cloud Export Types), either metadata only or flow logs and metadata.
3. In the IAM Role ARN field, enter the complete ARN of the IAM role created to grant Kentik’s AWS services access to the bucket (see Create an AWS Role). An IAM role ARN is structured as arn:aws:iam:: plus your AWS account number plus :role/ plus the name you gave to your role when you created it. For example: arn:aws:iam::012345678901:role/myLogsRole
   Note:
   - The ARN permissions must allow Kentik read access.
   - If write permissions are also included then Kentik will be able to delete logs once they've been exported.
4. Click the Verify Role button to confirm that Kentik can access the role.
5. From the AWS Region drop-down, choose the AWS region where the VPC instances that you wish to represent with this cloud reside.
6. Click the Verify Region button to confirm that Kentik can access the region.
7. If this cloud export includes both flow logs and metadata (see radio buttons above):
   - In the S3 Bucket Name field, enter the name of the bucket you created for this cloud export (see Create an S3 Bucket, e.g. test-logs-bucket).
   - Click the Verify Bucket button to confirm that Kentik can find the bucket from which to pull the logs for this cloud.
   - If you'd like Kentik to delete the logs after they've been exported, click the checkbox next to "Allow Kentik to delete logs from this bucket after ingesting them."
8. In the Cloud export name field, specify a name for the cloud export in Kentik.
9. From the Kentik billing plan drop-down, choose the Kentik billing plan to which this cloud should be assigned.
10. From the Sampling set of controls, choose the sampling rate for this cloud export (see Cloud Export Sampling).
11. Click the Save button to save the settings for the new cloud export and register it in Kentik. You'll be returned to the Public Clouds Page, where the new cloud export should now show in the Cloud Exports list.

The following settings determine the sampling rate for this cloud export:

• Sampling type:
  - Legacy: A maximum of 10k flows are randomly sampled per file.
  - Sampling Rate: Use the rate specified in the Sampling Rate field.
  - Unsampled: No sampling (all flow logs are sent).
• Sampling rate (present only when Sampling Type is Sampling Rate): Enter the sampling rate in the form of 1:N. Value must be between 2 and 2000.

The Automated Configuration tab gathers the information needed for Terraform to configure your AWS setup. The tab consists of the following parts, organized as numbered steps:

• Settings (steps 1 and 2)
• Configuration (textboxes in steps 3 and 4)

Note: If you configure a cloud export via the Automated Configuration tab but the Automatically register cloud export switch is off (see Automated Configuration Options) then Terraform will take care of the flow logging configuration process within AWS but you will still need to register the cloud export in Kentik via the Manual Configuration tab.

To create and run a configuration for Terraform:

1. In the Select your region drop-down (step 1), choose the AWS region where your organization has located the cloud resources that you wish to monitor. The region will be entered into the region field in the configuration shown in step 3.
2. In the Select options form (step 2), make the settings covered in Automated Configuration Options, which change the configuration shown in step 3.
3. Copy the generated Terraform configuration (step 3) and save it to a file named main.tf in an empty directory in the environment where you are running Terraform.
4. In the CLI of the environment where you are running Terraform, run the four commands shown in step 4.
5. Click the Finish button, which takes you to the Settings » Public Clouds page (see Public Clouds Page) where you will see the new cloud export listed in the Cloud Exports list.

The Terraform configuration template in step 3 is completed as you make the following settings in step 2:

• Enable flow logs:
  - For all VPCs in the selected region(s): The automation stack will query AWS for a list of all of the VPCs in the selected region and then automatically configure flow logs for all of them.
  - For selected VPCs in the selected region(s): Use when you don’t want flow logs on every VPC or flow logs are already configured on some of your VPCs. If this option is selected you will need to manually enter a list of VPC IDs as the value of the vpc_id_list parameter in the Terraform configuration (textbox in step 4).
• Write logs to bucket: A drop-down from which you choose a flow logging interval:
  - Every minute (recommended): Produces a greater volume of logs but at a more constant and smooth rate. This is usually preferable for most use cases, such as traffic engineering, security, and real time monitoring.
  - Every 10 minutes (AWS default): Reduces log volume and therefore reduces AWS charges.
• Automatically create necessary role in AWS account: This option enables your organization to forego the convenience of having the AWS role created automatically if that would be considered contrary to your security protocols.
• Automatically register cloud exports:
  - If on, the Cloud Export Configuration fields will be displayed under Select options and Terraform will automatically create the values needed to enable a Kentik cloud export from an S3 bucket in AWS.
  - If off, Terraform will still take care of the flow logging configuration process within AWS but you will then need to register the cloud export in Kentik via the Manual Configuration tab.
• Use External ID: If on, the cloud export configuration will include a Kentik-provided ID that will enable 3^rd party access to an S3 bucket for the purpose of exporting collected flow logs (see AWS documentation at https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user_externalid.html).

The following fields enable Terraform to automatically register a cloud with Kentik:

• Cloud Export Name Prefix: Specify an identifying string to attach to the cloud export name in Kentik.
• S3 Bucket Prefix: Specify an identifying string to attach to the S3 bucket name.
• IAM Role Prefix: Specify an identifying string to attach to the IAM role.
• Billing Plan: Choose the Kentik billing plan to which this cloud should be assigned.

Note: The prefix fields above enable you to add to the export name, bucket, and role any string (e.g. account-ID_region-name) that will help you to later identify which cloud export they are related to. You need not use the same prefix for all three fields.

Kentik supports ingest of flow logs from one or more cloud resource (e.g. virtual machine) in Microsoft Azure. The setup of a cloud export from Azure is mostly performed in the Kentik portal but you'll need information from your Azure configuration, and for certain steps you will be taken to the Azure console for authorization and to enter information. The steps are detailed in Kentik for Azure, which also includes background information about AWS flow logs (see About Azure Flow Logs) and step-by-step instructions for the parts of the setup process that are performed in Azure itself.

The Kentik v3 and v4 portals both support ingest of flow logs from one or more Google Cloud Platform VPC. The setup process in v3 is detailed in Kentik for GCP, which also includes step-by-step instructions for the parts of the setup process that are performed in GCP itself. These instructions apply as well to the steps in v4 cloud onboarding that are performed in GCP.

In v4 onboarding, the GCP cloud registration workflow is composed of the following steps:

1. Enable Flow Logs: In GCP, enable flow logs for each VPC subnet that you’d like to cover with Kentik (see Enable VPC Flow Logs). When done, come back to the Kentik workflow and click Next.
2. Project: To initiate logging in GCP you'll create a "sink" that specifies the destination to which logs will be exported (see Create a New Topic). In this Kentik workflow step you'll enter the name of the GCP project in which you created the sink, then click Next.
3. Topic & Subscription: Complete basic setup of logging in GCP, then click Next:
   - Create a New Topic: Configure Google's Stackdriver Logging to export flow logs to a new Pub/Sub topic (see Create a New Topic).
   - Create a Pull Subscription: Enable Kentik to initiate requests to the Cloud Pub/Sub server so we can retrieve flow logs from the newly created topic (see Create a Pull Subscription).
   - Enter Subscription Name: Enter the name of the newly created subscription.
   - Grant Kentik Read-Permission: Permit Kentik to access the subscription from the Google Cloud account on which we receive flow logs. To do this, we add the Kentik account to the subscription as a member (see Set Permissions).
4. BGP Peering: Choose whether Kentik should enrich the traffic data from this cloud with BGP by assigning to it the BGP table from one of your physical devices (e.g. a router in a data center) that is already set to peer with Kentik. If so, you'll enable BGP for the cloud and specify the master device.
5. Enter Details: Enter the name by which you want this cloud to appear in the Kentik portal, as well as a description.
6. Verify Details: This summary page allows you to confirm your settings for this cloud:
   - Review the listed settings: Click the Edit link for any setting that isn't correct; you'll be taken back to that step of the workflow, where you can make changes. Then click Next as needed to get back to the summary page.
   - Register the cloud: When all settings are correct, click the Create GCP Cloud button, which will take you to the individual device status page for the newly registered cloud.

The overall workflow for creating a cloud export to monitor OCI resources in Kentik is outlined in OCI Logging Setup Overview. The process involves two main phases:

1. In the OCI console, enable flow logging on the resources that you'd like to monitor in Kentik; see Configure Log Export in OCI.
2. On the Monitor your OCI Cloud page, complete the settings described in Add Cloud Export in Kentik.

When you've successfully worked through the above procedures you'll be returned to the Public Clouds page, where the new cloud export from OCI will be present in the Cloud Exports List. All resources (VCNs, etc.) that contribute logs to the OCI Object Storage bucket that is the source of the cloud export are typically represented in Kentik as a single device that is listed in the Devices list on the export's Cloud Details Page.

Note: For questions about the above procedure, contact Kentik (see Customer Care).