Kentik for Azure

The export of flow logs from Microsoft Azure, which enables a unified view of network traffic data in hybrid cloud environments, is covered in the following topics:

• About Azure Flow Logs
• Azure Logging Setup Overview
• Gather Azure Information
• Configure Export in Kentik

The basics of Azure flow logs are covered in the following topics:

• Azure Flow Log Overview
• Azure Flow Log Formats
• Azure Flow Log Retention
• Azure Flow Log Resources

Microsoft Azure is a cloud computing service created by Microsoft for use alone or in a hybrid configuration in which Azure resources work with other cloud resources and/or with traditional data centers (as shown below). With the introduction of support for Azure flow logs, Kentik now enables network traffic data from Azure resources to be used alongside data from other supported sources — both physical and cloud — for visualization, monitoring, alerting, and analytics.

In Azure, flow logging is a service provided by Azure Monitor, which has the resource provider namespace Microsoft.Insights. Flow logs are generated by network security groups. Each NSG represents resources in your Azure subscription, such as virtual machines (VMs), on which network traffic comes (ingress) and goes (egress). The flow logs, structured as JSON, provide the flow data detailed in the Azure documentation topic Log file.

Kentik does not consume flow logs directly from the monitored resources. Instead, the log files are exported to a storage account that includes the logs generated by all NSGs that share a given location and have been assigned to the same resource group.

A storage account is accessed by Kentik using NSG Flow Exporter (a Kentik-built enterprise application for Azure), which forwards the flow logs to KDE (see About Kentik Data Engine). The KDE ingest layer then normalizes the logs to Kentik's internal records format, enriches each record with Azure-specific details, and stores the resulting records in KDE for access via Kentik.

The logs ingested by Kentik for a given storage account are represented in Kentik as a single "cloud export" with a single "cloud device" (see Exports and Devices in Azure). The sampling rate for each cloud export is set independently (see Cloud Export Sampling).

The JSON flow records created by Azure flow logging cover all inbound and outbound IP flows for each NSG rule and include the flow's network interface (NIC), 5-tuple information, traffic decision, and throughput information (Version 2 only). For flow logs to be represented correctly in the Kentik portal their format must be NSG version 2. For details, see the following Azure documentation topics:

• How logging works
• Log format
• Sample log records

Deletion of flow logs after they've been ingested by Kentik enables you to minimize the costs associated with log data retention in the cloud. In Azure, log deletion is controlled by a retention setting that determines how long after its creation each log is kept. The way the retention duration is set depends on the method you choose to configure log export (see Choose Configuration Method):

• If you configure via script (recommended), the default value for log retention will be two days. You can change this setting by modifying the script as described in step 4 of Generate PowerShell Script.
• If you configure manually, you can change the Retention setting in Azure's Flow logs settings dialog, which you'll be directed to by the instructions provided when you click the Configure Manually button in the Configure Flow Log Export tile of Kentik's Monitor your Azure Cloud page.

Note: In Azure, retention is specified as an integer representing whole days, with a valid range of 1 -365.

For further information about Azure, please refer to the following Microsoft Azure documentation:

• Information about Azure Monitor: Azure Monitor log data documentation.
• For more about Azure flow logging, see Introduction to NSG Flow Logging.
• For information on the terminology Microsoft uses in reference to Azure, see the Azure Glossary.

To bring Azure logs into Kentik you'll need to create a cloud export (see Cloud Exports and Devices) in the Kentik portal. The workflow for creating and configuring the cloud export is as follows:

• In your Azure portal, gather the information that you'll need later in Kentik (see Gather Azure Information).
• On the Monitor your Azure Cloud page, specify the Azure resources from which you want to export logs (Define Storage Account).
• On the Monitor your Azure Cloud page, authorize Kentik's Azure application, NSG Flow Exporter, to access the Azure directory containing the resources from which you want to export flow logs to Kentik (Define Storage Account).
• On the Monitor your Azure Cloud page, specify the resource groups from which you want to apply data enrichment (optional).
• On the Monitor your Azure Cloud page, generate a PowerShell script based on the information provided in earlier steps (Generate PowerShell Script).
• In the Azure portal, use the script to configure Azure export settings (Configure Using PowerShell).
• On the Monitor your Azure Cloud page (Kentik portal), complete general settings for the cloud export (Name This Cloud Export).

Successful completion of the tasks listed in the workflow above will have the following effect in the Kentik portal:

• A new cloud export will be shown as an added row in the Kentik portal’s Cloud Exports list (Settings » Public Clouds; see Cloud Exports List). The cloud export will represent the collection of Azure resources whose logs are pulled by Kentik from one Azure storage account.
• The Devices column in the Cloud Exports list will show one cloud device.
• The Cloud Devices List on the Details page for the new cloud device will list one device, the name of which will match the name you give to the storage account.

Adding an Azure cloud export in Kentik involves an exchange of information between Kentik and the Azure portal. To complete the setup process, you'll need to gather information from your account in the Azure portal (https://portal.azure.com) as covered in the following topics:

• About Azure Roles
• Check Azure Role
• Role to Enable Exporter
• Role to Manage Exports
• Find Azure Subscription ID
• Find Resource Group and Location

Managing Kentik integration with Azure involves two distinct types of roles:

• Required only once, at the start of the setup process for each tenant, the Global Administrator role is used to allow you to grant the permissions needed for Kentik NSG Flow Exporter, which is a Microsoft-registered Enterprise Application, to access the resources (e.g. your "default directory") from which you'll be sending flow logs and metadata to Kentik. To set up the Global Administrator role, see Role to Enable Exporter.
• Once the required permissions have been granted to NSG Flow Exporter by a Global Administrator, a user with any role can manage the setup and maintenance of log exports. To set up to manage exports, see Role to Manage Exports.

Note:
- To check what role you currently have, see Check Azure Role.
- To see if other roles in your organization (e.g. Application Administrator or Cloud Application Administrator) may also be able to grant the permissions required for NSG Flow Exporter, please refer to the Azure document Available roles.

To check whether you are a Global Administrator or other qualifying role:

1. On the Home page of your Azure portal, click Azure Active Directory in the sidebar at left.
2. In the resulting Default Directory - Overview page, find the Manage list (second sidebar from the left) and click Roles and administrators.
3. Your role will be indicated at the top of the main page (above the Administrative roles heading).

If Kentik NSG Flow Exporter hasn't already been granted the required permissions in your tenant, and you're not already a Global Administrator (or other qualifying role), you'll need to be made one by an administrator of your Azure account. If you are one, you may wish to designate other Azure users so that they can also enable the exporter. Either way, the process is as follows:

1. In the Administrative roles list (main display area) of the Roles and administrators page, click Global Administrator.
2. On the resulting Global administrator - Members page, you'll see a list of users who are global administrators. Above the list, click Add member.
   Note: If you're not a global administrator this option will be grayed out.
3. In the resulting Add Member popup you'll see a list of members. Find the user(s) that you want to designate as a global administrator (use the Select field to filter the list) and click them to add them to the Selected members list.
4. Click the Select button at the bottom of the popup. The popup will close and the selected members will now be present in the list of global administrators. Anyone in that list who is also registered as a user in Kentik should be able to add an Azure cloud export.

Note: For further information on the assignment of administrator roles, please refer to the Azure documentation View and assign roles.

Once Kentik NSG Flow Exporter has been granted the required permissions by a Global Administrator, other users in the same tenant can manage setup of the collection and storage of flow logs and metadata related to the following.

• Metadata related to cloud naming, routing, NSG ACL, etc.
• Storage locations containing NSG flow logs and firewall logs.

To do so, a user would do the following:

1. Create a "Kentik" role with read-only permissions for the above data and locations.
2. Assign the Kentik NSG Flow Exporter application to the role.
   Note: For high-level views of traffic paths from on-premises to cloud resources, most organizations assign the application at a Tenant level.
3. In the resulting Add Member popup you'll see a list of members. Find the role created (use the Select field to filter the list) and click the “Kentik NSG Flow Log Exporter” to add them to the Selected members list.
4. Click the Select button at the bottom of the popup. The popup will close and the selected members will now be present in the list of global administrators. Anyone in that list who is also registered as a user in Kentik should be able to add an Azure cloud export.

Assets in Azure are each associated with a subscription. To add a cloud export, you'll need to find the subscription ID for the Azure directory containing the assets from which you want to export flow logs:

1. On the Home page of your Azure portal, click All Services in the sidebar at left.
2. At the left of the main All Services page, click General to filter the list of services.
3. In the list of General services, click on Subscriptions.
4. A table at the bottom of the resulting Subscriptions page will list all of your subscriptions. Copy and save the 32-digit GUID in the Subscription ID row of the subscription associated with the assets from which you want to send flow logs to Kentik.

A cloud export in Kentik represents a set of assets in Azure that share a unique Storage Account. To create a Storage Account you'll need to find the Resource Group and Location for the assets:

1. On the Home page of your Azure portal, click Virtual Machines in the sidebar at left.
2. In the table on the resulting page, find one of the VMs from which you want to export flow logs.
3. Copy and save the values in the Resource Group and Location columns.

The configuration in Kentik of a cloud export from Azure is covered in the following topics:

• Starting Azure Export Setup
• Define Storage Account
• Define Enrichment Scope
• Choose Configuration Method
• Generate PowerShell Script
• Configure Using PowerShell
• Azure Script Operations
• Name This Cloud Export

Once you have the required information from the Azure portal, you're ready to start adding an Azure cloud export in Kentik. To export flow logs from assets in Azure, you'll use the multi-step wizard to specify your Azure resources and grant permission for NSG Flow Exporter to access the Azure directory containing those assets. To grant permissions for NSG Flow Exporter, the wizard automatically checks and authorizes Azure to create a "service principal" representing NSG Flow Exporter. This is done in the background as you complete step 1 of the multi-part wizard.

Begin by opening the Monitor your Azure Cloud page, which is structured as a multi-step wizard that you work through to configure a cloud export:

1. Click Settings in the main portal menu, then Public Clouds on the Settings page
2. On the Public Clouds page click the Add Azure Cloud button to open the Monitor your Azure Cloud page.
3. Step 1 of the wizard is the Define Storage Account pane. Use the Subscription ID field to enter the ID of the subscription associated with the Azure directory containing the assets (see Find Azure Subscription ID).
4. Use the Resource Group to Monitor field to enter the resource group that you gathered in Find Resource Group and Location. To verify that permission has been granted to access all of the required APIs for the resource group, click the Verify button.
5. From the Location drop-down, choose the location that you gathered in Find Resource Group and Location.
6. In the Storage Account Name field, enter a name for the storage account to which logs will be exported from the above-specified Azure resources. The name must not be already in use by any other storage account, whether in your subscription or that of another Azure user.
   Note: Kentik must be able to access your storage account from the following public Azure IPs: 20.69.189.228 and 20.69.185.115.
7. Turn on the Flow Log Collection switch.
8. From the Sampling set of controls, choose the sampling rate for this cloud export (see Cloud Export Sampling).
9. Turn on the Firewall Log Collection switch.
10. Turn on the Metrics Collection switch to enable historical collection of cloud provider metrics (see Historical Metrics Collection).
11. Click Next to authorize access to Azure, which is required for NSG Flow Exporter to export flow logs from assets in Azure. The page will refresh and a popup will appear confirming that the Authorization Completed Successfully, meaning that Kentik NSG Flow Log Exporter has been added into the customer Active Directory structure.
12. Click Next to proceed to the next step of the wizard.

The following settings determine the sampling rate for this cloud export:

• Sampling type:
  - Sampling Rate: Use the rate specified in the Sampling Rate field.
  - Unsampled: No sampling (all flow logs are sent).
• Sampling rate (present only when Sampling Type is Sampling Rate): Enter the sampling rate in the form of 1:N. Value must be between 2 and 2000.

Kentik supports the collection of historical metrics for Azure. Along with real-time metrics, this data can provide a comprehensive view of performance trends and patterns over time. The collection of historical metrics must be enabled in the configuration of the cloud export (see Azure Provider Settings). Follow the steps below to enable historical metrics for your Azure cloud export:

• On the Kentik portal page, go to the Public Clouds page (Settings » Public Clouds).
• In the Cloud Exports list, click the edit icon on the Azure cloud export that you would like to edit. This will open the cloud export settings dialog.
• Enable the switch for Metrics Collection.
• Click Save to save and exit the dialog.

Once historical metrics have been enabled, you can view the metrics with the following steps:

• On the Kentik portal page, go to the Metrics Explorer page (Settings » Network Monitoring System » Metrics Explorer).
• Click Query in the SubNav to open the Metrics Explorer Query Sidebar.
• Within the Measurement pane, you can select the different available metrics using the drop-down.

The next step, which is optional, is to configure the network scope for data enrichment, which enables Kentik to correlate your organization's flow data with additional information such as GeoIP and BGP:

1. In the Subscription IDs box, paste or drag a file of comma-delimited subscription IDs to view the resource groups associated with the subscriptions.
   Note: The Subscription ID must not be in use by any other company.
2. A list of the inputted Subscription IDs will populate the page. Each subscription ID contains an All Resource Groups drop-down that lists all the resource groups within the subscription. Select the desired resource groups for enrichment.
   - The All Resource Groups drop-down is disabled if the subscription is in use by any other user.
   - The Subscription IDs box displays the number of valid (green checkmark) and invalid (red exclamation) subscriptions.
   - Repeat the previous step to add additional subscriptions to the list.
   - Use the Remove button to remove subscriptions from the list.
3. Click Next to proceed to the next step of the wizard.

In step 3 of the Monitor your Azure Cloud wizard, Configure Flow Log Export, you'll configure your Azure instance to export flow logs from the specified resource group and location to an Azure storage account. You can choose either of the following two methods, each of which has their own tab on the page:

• Configure using PowerShell (recommended): The textbox on this tab contains a Kentik-generated script that is based on the information provided in Define Storage Account. You run the script in the Azure portal's PowerShell instance as described in Configure Using PowerShell.
• Configure Manually: The Configure Manually tab contains a set of steps that you can execute in the Azure portal.
  Note: To get a feel for the process involved in manual configuration, see the Microsoft Azure document Tutorial: Log network traffic.

If you're on the Configure Using PowerShell tab you'll see a textbox that contains a Kentik-generated PowerShell script.

1. Check that the values for subscription ID, resource group, storage account, and location that are listed near the top of the dialog are correct. If not, click the Back button at the bottom of the page until you return to step 1 and correct the information that you entered in at Define Storage Account.
2. Click the Copy to Clipboard button at the top right of the textbox.
3. If you'd like the logs to be retained for a duration other than the default two days:
   - Paste the script into a text editor.
   - Modify the value of the RetentionInDays argument in the declaration of the $ret variable (currently on line 220). The new value must be an integer, representing whole days, between 1 and 365.
   - Copy the edited script back to the clipboard.

To configure Azure log export in the Azure portal using PowerShell:

1. Navigate to your Azure portal and log in.
2. In the main Azure navbar, click the PowerShell icon (>_).
3. Click PowerShell in the popup
4. PowerShell will open and initialize.
5. When initialization is complete, type code at the prompt. The code editor will open.
6. Paste the script from Generate PowerShell Script into the code editor, then save the script:
   - Right click anywhere in the editor and select Save to open the Save new file dialog.
   - Enter a name for the script. The name must end with.ps1 extension, e.g. MyKentikScript.ps1.
   - Click the Save button.
7. Choose Close from the script editor menu to close the editor.
8. Back in PowerShell, enter the full path to the script at the prompt, e.g.:
   /home/user_name/MyKentikScript.ps1
   Note: The user_name for an Azure script file path is the first word of the full user name, e.g. if your user name is "Sallie Mae" then sallie.
9. The script will run (see Azure Script Operations). When it's done you'll see a confirming message with the information that you entered for subscription ID, resource group, location, and storage account.

Note: PowerShell instances are ephemeral. If you are timed out or otherwise lose connection during the procedure above, you must re-start the above process from step 2.

The configuration script executes a series of operations that configure Azure for log export based on your settings in the Monitor your Azure Cloud page. These operations include the following:

• Confirm existence of the provided subscription ID.
• Confirm existence of the specified location.
• Confirm existence of the specified resource group.
• Confirm that a service principal has been created in your Azure instance for the Kentik application NSG Flow Exporter.
• Confirm that NSG Flow Exporter has been granted the required access ("Reader") to the specified resource group.
• Create a storage account with the specified name for the specified resource group and location.
• Grant NSG Flow Exporter the required access ("Contributor") to the storage account.
• Confirm that the Network Watcher feature is registered in your subscription.
• Confirm that Network Watcher exists for the specified resource group and location.
• Confirm that the specified subscription is registered with Microsoft.Insights, the resource provider namespace for Azure Monitor, which provides the logging resource.
• Build a list of network security groups (NSGs) found in the specified resource group and location.
• Enable v2 NSG flow logs for each found NSG.

Once your Azure instance is configured (either manually or with PowerShell) to export flow logs to a storage account from the specified resource group and location, you'll return to Kentik to complete the final step of the wizard:

1. If you're on the Configure Flow Log Export step, click the Next button, which will take you to step 4, Name this Cloud Export.
2. In the Name field, enter the name that should be shown for the cloud export in the Cloud Exports List on the Public Clouds page.
3. In the Description field, briefly describe the Azure resources covered by this cloud export.
4. From the Billing Plan drop-down, choose the Kentik billing plan to which the cloud export should be assigned (see About Plans).
5. Click Save to complete the wizard. An Added Successfully popup will appear to confirm completion. You may now add additional cloud exports or Cancel out of the wizard.
6. The cloud export can be viewed in the Cloud Exports List on the Public Clouds page.