Kentik for Azure

This guide provides instructions for integrating Kentik with Microsoft Azure.

Important Note on Azure Flow Logs

Kentik supports processing both VNet (Virtual Network) flow logs and NSG (Network Security Group) flow logs from your Microsoft Azure storage account. However, Microsoft advises enabling only one type of flow log at a time to prevent duplicate traffic recording and additional costs.

IMPORTANT: Kentik encourages all customers to use Microsoft’s migration scripts to enable VNet flow logs. This ensures your Kentik account remains current and avoids potential issues.

For more information, refer to the following Azure documentation:

• VNet Flow Logs Overview

• NSG Flow Logs Migration

Process Overview

Integrating Azure with Kentik involves setting up both the Azure environment and the Kentik portal to collect metadata, flow logs, firewall logs, and metrics from VNets or NSGs. Here's how the process works:

1. Logging Setup (Azure):

   1. Activate flow logs for desired VNets/NSGs and set up log export to a storage account.

   2. Create and grant access for Kentik to pull logs from the storage account, using either the pre-integrated Kentik enterprise application or a custom application registration (Service Principal Name) for secure authentication.

2. Create a Kentik Cloud Export:

   1. Configure a new "cloud export" in the Kentik portal to ingest data from Azure.

      Note: When using the Powershell Automated Configuration option, additional steps in the Azure portal are required while creating the Kentik cloud export.

Once the setup is complete, you can use the Kentik portal to:

• Monitor your Azure network traffic.

• Visualize resource utilization.

• Gain insights for optimizing network performance and enhancing security monitoring.

Note: See the Cloud Overview for an introduction to Kentik cloud setup.

About Azure Flow Logs

The basics of Azure flow logs are covered in the following topics.

Azure Flow Log Overview

Microsoft Azure provides a cloud computing service that can operate independently or in a hybrid setup with other cloud resources and traditional data centers. Kentik supports Azure flow logs, allowing network traffic data from Azure to be integrated with other sources for visualization, monitoring, alerting, and analytics.

Key Points:

• Flow Logging Service:

  • Provided by Azure Monitor under the namespace Microsoft.Insights.

  • Flow logs are generated by VNets or NSGs in your Azure subscription.

  • Logs capture ingress and egress traffic and are structured in JSON format (see Azure docs on Log format).

• Log Export Process:

  • Logs are not consumed directly from resources. Instead, they are exported to a storage account that aggregates logs from NSGs/VNets in the same location and resource group.

NSG Flow Logs vs. VNet Flow Logs

Azure supports the collection of both VNet flow logs and NSG flow logs, as compared here:

• NSG Flow Logs: Capture IP traffic flowing through an NSG.

  • NSGs are used to filter network traffic to and from Azure resources in a virtual network.

  • NSG flow logs provide detailed information about each flow, including source and destination IP addresses, ports, protocol, and whether the traffic was allowed or denied by the NSG rule.

  • Offer granular visibility at the NSG level, useful for understanding security group efficacy and troubleshooting access issues for specific subnets or VMs.

• VNet Flow Logs: Capture traffic entering and leaving the VNet.

  • Ideal when managing complex network topologies with many NSGs.

  • Simplify troubleshooting and security monitoring for the entire virtual network.

Azure Flow Log Formats

Note: Kentik recommends following Microsoft’s guidance to use VNet flow logs instead of NSG flow logs going forward (see Microsoft's flow logging recommendation).

Azure flow logs, structured in JSON, capture all inbound and outbound IP flows for each VNet or NSG rule. Key components of these logs include:

• Network Interface (NIC): Information about the network interface involved in the flow.

• 5-tuple Information: Details such as source and destination IP addresses, source and destination ports, and protocol.

• Traffic Decision: Whether traffic was allowed or denied

• Throughput Information: Provides data on the volume of traffic.

For more insights into the structure and content of Azure flow logs, see the following Azure documentation topics:

• How logging works

• Log format

• Sample log records

Azure Flow Log Retention

Managing the retention of Azure flow logs is crucial for minimizing cloud storage costs. Here’s how retention is handled:

• Retention Setting: Determines how long each log is kept after creation. It can be configured in two ways:

  • Via Script (Recommended):

    • Default retention is set to two days.

    • Modify the retention duration in the script as described in step #4 of Generate PowerShell Script.

  • Manual Configuration: Adjust the Retention setting in Azure’s Flow logs settings dialog.

• Retention Range:

  • Specified as an integer in whole days.

  • Valid range is from 1 to 365 days.

Azure Flow Log Resources

For more detailed information on Azure and its flow logging capabilities, refer to the following Microsoft Azure documentation:

• Azure Monitor log data documentation

• Introduction to NSG Flow Logging

• Azure Glossary

Logging Setup (Azure)

The logging setup tasks in the Azure Portal are covered in the following topics.

About Azure Roles

Integrating Kentik with Azure requires understanding two key Azure user roles:

• Role to Enable Exporter:

  • Typically, an Application Administrator role is used at the start of the setup process for each tenant.

  • The way you grant these permissions depends on whether you use the pre-integrated Kentik enterprise application or a Custom app registration.

    • Kentik Enterprise Application: When you select this option in Kentik, you'll grant permissions to a pre-existing enterprise application. This is typically done through the Microsoft Entra ID » Enterprise applications panel, where you search for the "Kentik NSG Flow Exporter" application and assign it the necessary roles to access your resources.

    • Custom App Registration: When you select this option, you are responsible for creating the application identity in your Azure tenant. The key difference is that you will grant the application permissions from the resource's perspective rather than from the application's perspective. After creating the app registration, you navigate to the specific Azure resource (e.g., a subscription or a resource group) and use the Access control (IAM) panel to assign a role to your custom app's Service Principal.

• Role to Manage Exports:

  • After permissions are granted by an Application Administrator, any user role can manage the setup and maintenance of log exports.

Notes:

• Use Check Azure Role to verify your current role.

• For additional roles that may grant permissions for NSG Flow Exporter, see the Azure document Available roles.

Check Azure Role

To verify your role in Azure, follow these steps:

1. On the Home page of your Azure portal, click Microsoft Entra ID in the sidebar at left.

2. In the resulting Default Directory - Overview page, find the Manage list (second sidebar from the left) and click Roles and administrators.

3. Your role will be indicated at the top of the main page (above the Administrative roles heading).

Role to Enable Exporter

To grant the necessary permissions, follow the steps below based on your selected authentication method (see About Azure Roles).

Kentik Enterprise Application

To grant permissions for Kentik’s NSG Flow Exporter, follow these steps if you are an Application Administrator or need to designate one:

1. In the Administrative roles list on the Roles and administrators page, click Application Administrator.

2. On the Application administrator - Members page, click Add member above the list of current administrators.

   Note: This option will be grayed out if you are not an application administrator.

3. In the Add Member popup, find and select the user(s) you want to designate as application administrators.

4. Use the Select field to filter the list, then click a user to add it to the Selected members list.

5. Click the Select button to close the popup and return to the list of application administrators.

Note: For more details on assigning administrator roles, refer to the Azure documentation View and assign roles.

Custom Application Registration

If you chose the custom app registration option, you will grant permissions to your own application's Service Principal. This process is managed from the resource's perspective in the Azure portal.

1. Navigate to the Azure resource you want to monitor (e.g., a specific resource group or your entire subscription).

2. In the resource's menu, select Access control (IAM).

3. Click the Add button and then select Add role assignment.

4. Choose the required role (e.g., Reader).

5. On the Members tab, select User, group, or service principal.

6. In the Select members field, search for the name of your custom app registration.

7. Select your application from the list and click Review + assign.

Role to Manage Exports

After the Service Principal (either the Kentik NSG Flow Exporter or your Custom app registration) has been granted permissions by an Application Administrator, other users can manage the setup for collecting and storing flow logs and metadata. Here’s how to proceed:

1. Create a "Kentik" role with read-only permissions for metadata related to cloud naming, routing, ACL, and storage locations for flow logs and firewall logs.

2. Assign the Service Principal representing the application to this role.

   Note: For comprehensive traffic path views, it’s common to assign the application at a Tenant level.

3. In the Add Member popup, locate the created role using the Select field to filter the list.

4. Click the “Kentik NSG Flow Log Exporter” or the name of your custom app registration to add it to the Selected members list.

5. Click Select to close the popup and return to the list of application administrators, which now includes the selected members.

Find Azure Subscription ID

To locate the subscription ID for exporting flow logs from Azure:

1. On the Azure portal Home page, click All Services in the sidebar at left.

2. Click General on the left of the All Services page to filter the list.

3. Click on Subscriptions in the list of General services.

4. In the table at the bottom of the Subscriptions page, find and copy the 32-digit GUID in the Subscription ID row for the relevant subscription.

Find Resource Group and Location

To identify the Resource Group and Location for creating a Storage Account:

1. On the Azure portal Home page, click Virtual Machines in the left sidebar.

2. In the resulting table, find the VM from which you want to export flow logs.

3. Copy and save the values in the Resource Group and Location columns.

Create a Kentik Cloud Export

To configure an Azure cloud export in the Kentik portal:

1. Go to Settings » Public Clouds in the main nav menu.

2. Click Create Cloud Export to start the configuration wizard.

3. Choose Azure Cloud under Provider and Features.

4. Under Observability Features, select the data types to collect:

   1. Metadata collection (required): Automatically selected.

   2. Flow log collection: Select to collect flow logs.

   3. Help me configure my provider via Powershell: Choose this to receive a Kentik-generated Powershell script for automatic configuration in Azure Cloud Shell, see Automated Configuration.

   4. Firewall Collection: Select to collect Azure firewall logs.

   5. Cloud metrics history: Select to collect Azure metrics with Kentik’s NMS.

5. Click the green arrow to proceed.

6. Select an API access method by selecting either the Kentik enterprise application or Custom app registration tab.

   1. For Kentik enterprise application, no additional information is needed.

   2. For Custom app registration, enter the following information or choose from the Saved App Registrations (if any):

      1. Application (client) ID: Enter the ID for the custom application.

      2. Directory (tenant) ID: Enter the tenant ID.

      3. Select a credential from the Kentik Credential dropdown or click Create New Secret to create a new secret (see Credentials Settings Dialogs).

      4. Click Save App Registration.

7. Choose the location from the Location drop-down, as gathered in Find Resource Group and Location.

8. Enter the Subscription ID associated with the Azure directory containing the assets (see Find Azure Subscription ID).

9. Click Verify to authorize the Azure portal to create a Service Principal for Kentik's NSG Flow Exporter. Ensure your Azure role allows granting access to enterprise applications.

10. In the Resource Group to Monitor field, enter the resource group gathered in Find Resource Group and Location.

11. Click Verify Access to ensure permission has been granted to the resource group for all required APIs.

12. Enter a unique name in the Storage Account Name field for the storage account where logs will be exported.

    Notes:

    • Kentik must access your storage account from the following public Azure IPs: 20.69.189.228 and 20.69.185.115.

    • If you have a storage account in the WestUS2 region, use these IPs instead: 51.8.214.254 and 172.171.46.16 and contact your account team.

13. Under Sampling, choose from:

    1. Sampling Rate: After selecting this option, enter a sampling rate in the Sampling Rate field. The value must be between 2 and 2000.

    2. Unsampled: Select this option if you want all flow logs to be sent without sampling.

14. Click the green arrow to proceed.

15. Optionally, configure the network scope for data enrichment, see Define Enrichment Scope.

16. Click the green arrow to proceed

17. Enter the cloud export name/description or accept the defaults.

18. Select the appropriate Kentik billing plan for the cloud export from the Billing Plan dropdown.

19. Click Save to finalize the cloud export and return to the Public Clouds page, where the new export will be listed.

Define Enrichment Scope

To optionally configure the network scope for data enrichment, allowing Kentik to enhance your Azure flow data with additional information such as GeoIP and BGP, follow these steps:

1. In the Subscription IDs box, paste or drag a file containing comma-delimited subscription IDs. This will allow Kentik to view the resource groups associated with these subscriptions.

   Note: Ensure that the Subscription ID is not in use by any other company.

2. A list of the entered Subscription IDs will populate the page, each with an All Resource Groups drop-down listing all resource groups within that subscription. Select the desired resource groups for enrichment.

Notes:

• The All Resource Groups dropdown is disabled if the subscription is in use by any other user.

• The Subscription IDs box will indicate the number of valid (green checkmark) and invalid (red exclamation) subscriptions.

• You can repeat the process to add more subscriptions to the list.

• Click Remove to delete subscriptions from the list.

Automated Configuration

If you opted for Help me configure my provider via Powershell earlier, follow these steps. You'll use both the Kentik and Azure portals to complete the automated setup of an Azure cloud export.

Notes:

• As of February 2025, Kentik’s Powershell script can only be used for configuring VNet flow logs.

• For manual configuration guidance, see the Microsoft Azure document Tutorial: Log network traffic.

Generate PowerShell Script

To generate and customize a Powershell script from the Azure Powershell section of the Kentik wizard, follow these steps:

1. Verify Script Values:

   1. Check that the values for subscription ID, resource group, storage account, and location are correct.

   2. If any values are incorrect, click the gray left arrow until you return to the first wizard step and correct the entered information.

2. Click Copy to Clipboard at the top right of the textbox to copy the script.

3. Optional: To change the log retention duration from the default two days:

   1. Paste the script into a text editor.

   2. Locate the RetentionInDays argument in the declaration of the $ret variable (line 220).

   3. Modify the value to an integer representing whole days, between 1 and 365.

   4. Copy the edited script to the clipboard.

Configure Using PowerShell

To set up Azure log export using PowerShell in the Azure portal, follow these steps:

1. Navigate to your Azure portal and log in.

2. In the main Azure navbar, click the PowerShell icon (>_).

3. Click PowerShell in the popup and wait for PowerShell to initialize.

4. Once initialization is complete, type code at the prompt to open the code editor.

5. Paste and Save Script:

   1. Paste the script from Generate PowerShell Script into the code editor.

   2. Right-click anywhere in the editor and select Save to open the Save new file dialog.

   3. Enter a name for the script with a .ps1 extension (e.g., MyKentikScript.ps1).

   4. Click Save.

6. Choose Close from the script editor menu to close the editor.

7. From the PowerShell prompt, enter the full path to the script (e.g., /home/user_name/MyKentikScript.ps1).

   Note: The user_name for an Azure script file path is the first word of your full user name (e.g., "Sallie Mae" becomes sallie).

8. Execute the script (see Azure Script Operations). Upon completion, you'll see a confirming message with the details you entered for subscription ID, resource group, location, and storage account.

Note: PowerShell instances are ephemeral. If you time out or lose connection, restart the process from step #2.

Azure Script Operations

The configuration script for Azure log export performs the following operations based on your settings in the Kentik wizard:

• Confirmations:

  • Confirms the existence of the provided subscription ID, location, and resource group.

  • Confirms that a service principal has been created for the Kentik NSG Flow Exporter application.

  • Confirms that NSG Flow Exporter has been granted "Reader" access to the specified resource group and to enrich subscriptions and resource groups.

• Storage Account Setup:

  • Checks if a storage account exists for the specified name, resource group, and location and creates one if not.

  • Grants NSG Flow Exporter "Contributor" access to the storage account.

• Network Watcher and Insights Registration:

  • Confirms that a Network Watcher exists for the specified resource group and location.

  • Confirms that the specified subscription is registered with Microsoft Insights, the resource provider namespace for Azure Monitor.

• VNet Configuration:

  • Builds a list of VNets in the specified resource group and location.

  • Enables flow logs for each found VNet.

Using Your Cloud Export

Once the setup process is complete, you can view and utilize your cloud export in Kentik:

• Cloud Exports List:

  • Go to Settings » Public Clouds to see the updated list of cloud exports.

  • A new cloud export will be listed, representing the VNets or NSGs whose logs are pulled from the specified subscription.

• Devices Column:

  • Each VNet/NSG sending flow logs is listed as a cloud device.

  • Devices are named after their respective VPC subnet.

  • These names can be used as group-by and filter values in Kentik queries using the Device Name dimension.

• Metadata and Mapping:

  • The collected metadata, for example for subnets and gateways, enables Kentik to automatically map and visualize the topology of your Azure resources in the Kentik Map.

Historical Metrics Collection

Kentik allows for the collection of historical metrics for Azure. Along with real-time metrics, this data can provide a comprehensive view of performance trends and patterns over time.

The collection of historical metrics must be enabled in the configuration of the cloud export. Once enabled, you can view the metrics with the following steps:

1. From the Kentik portal main menu, go to Settings » Network Monitoring System » Metrics Explorer.

2. In the Measurement pane of the Metrics Explorer Query Sidebar, select a measurement starting with "/cloud/Azure/", then select from the available metrics using the dropdown. Click Run Query to execute the query.

Azure Endpoints Lists

Kentik needs permission to access selected Azure endpoints on your behalf in order to collect metadata and metrics, as detailed here.

Azure Metadata Endpoints

Entity type:

• subscriptions

• applicationGateways

• expressRouteGateways

• expressRouteCircuits

• azureFirewalls

• loadBalancers

• localNetworkGateways

• locations

• natGateways

• networkSecurityGroups

• p2SVpnGateways

• resourceGroups

• networkInterfaces

• networkWatchers

• flowLogs

• privateEndpoints

• publicIPAddresses

• routeTables

• virtualHubs

• hubVirtualNetworkConnections

• hubRouteTables

• virtualWans

• vms

• vmScaleSets

• vmScaleSetVMs

• vmScaleSetInterfaces

• vnets

• classicVnets

• vnetGateways

• vnetGatewayConnections

• vpnGateways

• vpnSites

• expressCircuitPeeringRouteTables