Kentik for Azure
The export of flow logs from Microsoft Azure, which enables a unified view of network traffic data in hybrid cloud environments, is covered in the following topics:
About Azure Flow Logs
The basics of Azure flow logs are covered in the following topics:
Azure Flow Log Overview
Microsoft Azure is a cloud computing service created by Microsoft for use alone or in a hybrid configuration in which Azure resources work with other cloud resources and/or with traditional data centers (as shown below). With the introduction of support for Azure flow logs, Kentik now enables network traffic data from Azure resources to be used alongside data from other supported sources — both physical and cloud — for visualization, monitoring, alerting, and analytics.
In Azure, flow logging is a service provided by Azure Monitor, which has the resource provider namespace Microsoft.Insights. Flow logs are generated by network security groups. Each NSG represents resources in your Azure subscription, such as virtual machines (VMs), on which network traffic comes (ingress) and goes (egress). The flow logs, structured as JSON, provide the flow data detailed in the Azure documentation topic Log file.
Kentik does not consume flow logs directly from the monitored resources. Instead, the log files are exported to a storage account that includes the logs generated by all NSGs that share a given location and have been assigned to the same resource group.
A storage account is accessed by Kentik using NSG Flow Exporter (a Kentik-built enterprise application for Azure), which forwards the flow logs to KDE (see About Kentik Data Engine). The KDE ingest layer then normalizes the logs to Kentik's internal records format, enriches each record with Azure-specific details, and stores the resulting records in KDE for access via Kentik. The logs ingested by Kentik for a given storage account are represented in Kentik as a single "cloud export" with a single "cloud device" (see Exports and Devices in Azure).
Azure Flow Log Formats
The JSON flow records created by Azure flow logging cover all inbound and outbound IP flows for each NSG rule and include the flow's network interface (NIC), 5-tuple information, traffic decision, and throughput information (Version 2 only). For details, see the following Azure documentation topics:
Azure Flow Log Retention
Deletion of flow logs after they've been ingested by Kentik enables you to minimize the costs associated with log data retention in the cloud. In Azure, log deletion is controlled by a retention setting that determines how long after its creation each log is kept. The way the retention duration is set depends on the method you choose to configure log export (see Choose Configuration Method):
- If you configure via script (recommended), the default value for log retention will be two days. You can change this setting by modifying the script as described in step 4 of Generate PowerShell Script.
- If you configure manually, you can change the Retention setting in Azure's Flow logs settings dialog, which you'll be directed to by the instructions provided when you click the Configure Manually button in the Configure Flow Log Export tile of Kentik's Monitor your Azure Cloud page.
Note: In Azure, retention is specified as an integer representing whole days, with a valid range of 1 -365.
Azure Flow Log Resources
For further information about Azure, please refer to the following Microsoft Azure documentation:
- Information about Azure Monitor: Azure Monitor log data documentation.
- For more about Azure flow logging, see Introduction to NSG Flow Logging.
- For information on the terminology Microsoft uses in reference to Azure, see the Azure Glossary.
Azure Logging Setup Overview
To bring Azure logs into Kentik you'll need to create a cloud export (see Cloud Exports and Devices) in the Kentik portal. The workflow for creating and configuring the cloud export is as follows:
- In your Azure portal, gather the information that you'll need later in Kentik (see Gather Azure Information).
- On the Monitor your Azure Cloud page, authorize Kentik's Azure application, NSG Flow Exporter, to access the Azure directory containing the resources from which you want to export flow logs to Kentik (Authorize Access to Azure).
- On the Monitor your Azure Cloud page, specify the Azure resources from which you want to export logs (Specify Azure Resources).
- On the Monitor your Azure Cloud page, generate a PowerShell script based on the information provided in earlier steps (Generate PowerShell Script).
- In the Azure portal, use the script to configure Azure export settings (Configure Using PowerShell).
- On the Monitor your Azure Cloud page (Kentik portal), complete general setting for the cloud export (Specify Export Settings).
- Back on the Monitor your Azure Cloud page, initiate validation of the configuration to complete registration of the cloud export (Validate Azure Setup).
Azure Clouds in the Portal
Successful completion of the tasks listed in the workflow above will have the following effect in the Kentik portal:
- A new cloud export will be shown as an added row in the Kentik portal’s Cloud Exports list (Settings » Public Clouds; see Cloud Exports List). The cloud export will represent the collection of Azure resources whose logs are pulled by Kentik from one Azure storage account.
- The Devices column in the Cloud Exports list will show one cloud device.
- The Cloud Devices List on the Details page for the new cloud device will list one device, the name of which will match the name you give to the storage account.
Gather Azure Information
Adding an Azure cloud export in Kentik involves an exchange of information between Kentik and the Azure portal. To complete the setup process, you'll need to gather information from your account in the Azure portal (https://portal.azure.com) as covered in the following topics:
Check Azure Role
To add an Azure cloud export to Kentik your role in Azure must allow you to grant the permissions that enable Kentik's NSG Flow Exporter application to access the resources (e.g. your "default directory") from which you'll be sending flow logs to Kentik. Kentik has confirmed that the role of "Global Administrator" works for this purpose.
Note: To see if other roles in your organization (e.g. Application Administrator or Cloud Application Administrator) may also be able to grant the required permissions please refer to the Azure document Available roles.
To confirm that you are a global administrator or other qualifying role:
- On the Home page of your Azure portal, click Azure Active Directory in the sidebar at left.
- In the resulting Default Directory - Overview page, find the Manage list (second sidebar from the left) and click Roles and administrators.
- Your role will be indicated at the top of the main page (above the Administrative roles heading).
If you're not a Global Administrator (or other qualifying role), you'll need to be made one by an administrator of your Azure account. If you are one, you may wish to designate other Azure users so that they can also add Azure cloud exports in Kentik. Either way, the process is as follows:
- In the Administrative roles list (main display area) of the Roles and administrators page, click Global Administrator.
- On the resulting Global administrator - Members page, you'll see a list of users who are global administrators. Above the list, click Add member.
Note: If you're not a global administrator this option will be grayed out.
- In the resulting Add Member popup you'll see a list of members. Find the user(s) that you want to designate as a global administrator (use the Select field to filter the list) and click them to add them to the Selected members list.
- Click the Select button at the bottom of the popup. The popup will close and the selected members will now be present in the list of global administrators. Anyone in that list who is also registered as a user in Kentik should be able to add an Azure cloud export.
Note: For further information on the assignment of administrator roles, please refer to the Azure documentation View and assign roles.
Find Azure Subscription ID
Assets in Azure are each associated with a subscription. To add a cloud export, you'll need to find the subscription ID for the Azure directory containing the assets from which you want to export flow logs:
- On the Home page of your Azure portal, click All Services in the sidebar at left.
- At the left of the main All Services page, click General to filter the list of services.
- In the list of General services, click on Subscriptions.
- A table at the bottom of the resulting Subscriptions page will list all of your subscriptions. Copy and save the 32-digit GUID in the Subscription ID row of the subscription associated with the assets from which you want to send flow logs to Kentik.
Find Resource Group and Location
A cloud export in Kentik represents a set of assets in Azure that share a unique Storage Account. All assets in the storage account must be in the same Resource Group and Location. To create a Storage Account you'll need to find the Resource Group and Location for the assets:
- On the Home page of your Azure portal, click Virtual Machines in the sidebar at left.
- In the table on the resulting page, find one of the VMs from which you want to export flow logs.
- Copy and save the values in the Resource Group and Location columns.
Authorize Access to Azure
Once you have the required information from the Azure portal, you're ready to start adding an Azure cloud export in Kentik. To export flow logs from assets in Azure, you'll need to grant permission for NSG Flow Exporter to access the Azure directory containing those assets. You'll do this by authorizing Azure to create a "service principal" representing NSG Flow Exporter.
Begin by opening the Monitor your Azure Cloud page, which is structured as a multi-step wizard that you work through to configure a cloud export:
- Click Settings in the main portal menu, then Public Clouds on the Settings page.
- On the Public Clouds page click the Add Azure Cloud button to open the Monitor your Azure Cloud page.
- Step 1 of the wizard is the Authorize Access to Azure pane. Use the Subscription ID field (shown at right) to enter the ID of the subscription associated with the Azure directory containing the assets (see Find Azure Subscription ID).
- Click the Next button, which will take you to the Azure portal.
- On the Azure login screen, pick the account associated with the assets from which you want to send flow logs to Kentik.
- Once you're logged in, you'll be taken to the Permissions requested screen (shown at right), where you'll be asked to grant the permissions required for NSG Flow Exporter to access resources in your account.
Note: If you are not taken to the Permissions requested screen, and are instead returned to the Kentik portal immediately after login, check that your Azure role allows you to grant the needed permissions (see Check Azure Role).
- Click the Accept button, which will create an Azure service principal for the NSG Flow Exporter application. You will then be returned to the Monitor your Azure Cloud page in the Kentik portal.
Specify Azure Resources
Flow logging requires two types of resources in Azure:
- The Azure resources (e.g. VM) for which you are generating logs.
- A "storage account" where the logs are collected and from which they are forwarded to Kentik (specifically to the Kentik Data Engine).
To set up log export, these resources must be identified to Kentik in step 2 of the cloud export wizard on the Monitor your Azure Cloud page. In the Specify Azure Resources pane:
- In the Resource Group to Monitor field, enter the resource group that you gathered in Find Resource Group and Location.
- From the Location drop-down, choose the location that you gathered in Find Resource Group and Location.
- Turn on the Enable Flow logs switch.
- In the Storage Account Name field, enter a name for the storage account to which logs will be exported from the above-specified Azure resources.
Note: The name must not be already in use by any other storage account, whether in your subscription or that of another Azure user.
- Click Next to proceed to the next step of the wizard.
Configure Azure Log Export
The next step is to configure your Azure instance to export flow logs, which is covered in the following topics:
Choose Configuration Method
In step 3 of the Monitor your Azure Cloud wizard, Configure Flow Log Export, you'll configure your Azure instance to export flow logs from the specified resource group and location to an Azure storage account. You can choose either of the following two methods, each of which has their own tab on the page:
- Configure using PowerShell (recommended): The textbox on this tab contains a Kentik-generated script that is based on the information provided in Specify Azure Resources. You run the script in the Azure portal's PowerShell instance as described in Configure Using PowerShell.
- Configure Manually: The Configure Manually tab contains a set of steps that you can execute in the Azure portal.
Note: To get a feel for the process involved in manual configuration, see the Microsoft Azure document Tutorial: Log network traffic.
Generate PowerShell Script
If you're on the Configure Using PowerShell tab you'll see a textbox that contains a Kentik-generated PowerShell script.
- Check that the values for subscription ID, resource group, storage account, and location that are listed near the top of the dialog are correct. If not, click the Back button at the bottom of the page and correct the information that you entered in the Specify Azure Resources step.
- Click the Copy to Clipboard button at the top right of the textbox.
- If you'd like the logs to be retained for a duration other than the default two days:
- Paste the script into a text editor.
- Modify the value of the RetentionInDays argument in the declaration of the $ret variable (currently on line 167). The new value must be an integer, representing whole days, between 1 and 365.
- Copy the edited script back to the clipboard.
Configure Using PowerShell
To configure Azure log export in the Azure portal using PowerShell:
- Navigate to your Azure portal and log in.
- In the main Azure navbar, click the PowerShell icon (>_). PowerShell will open and initialize.
- When initialization is complete, type code at the prompt. The code editor will open.
- Paste the script from Generate PowerShell Script into the code editor, then save the script:
- Using the drop-down menu ("..." icon) at the upper right of the script editor, open the Save new file dialog.
- Enter a name for the script. The name must end with.ps1 extension, e.g. MyKentikScript.ps1.
- Click the Save button.
- Choose Close from the script editor menu to close the editor.
- Back in PowerShell, enter the full path to the script at the prompt, e.g.:
Note: The user_name for an Azure script file path is the first word of the full user name, e.g. if your user name is "Sallie Mae" then sallie.
- The script will run (see Azure Script Operations). When it's done you'll see a confirming message with the information that you entered for subscription ID, resource group, location, and storage account.
Note: PowerShell instances are ephemeral. If you are timed out or otherwise lose connection during the procedure above, you must re-start the above process from step 2.
Azure Script Operations
The configuration script executes a series of operations that configure Azure for log export based on your settings in the Monitor your Azure Cloud page. These operations include the following:
- Confirm existence of the provided subscription ID.
- Confirm existence of the specified location.
- Confirm existence of the specified resource group.
- Confirm that a service principal has been created in your Azure instance for the Kentik application NSG Flow Exporter.
- Confirm that NSG Flow Exporter has been granted the required access ("Reader") to the specified resource group.
- Create a storage account with the specified name for the specified resource group and location.
- Grant NSG Flow Exporter the required access ("Contributor") to the storage account.
- Confirm that the Network Watcher feature is registered in your subscription.
- Confirm that Network Watcher exists for the specified resource group and location.
- Confirm that the specified subscription is registered with Microsoft.Insights, the resource provider namespace for Azure Monitor, which provides the logging resource.
- Build a list of network security groups (NSGs) found in the specified resource group and location.
- Enable v2 NSG flow logs for each found NSG.
Specify Export Settings
Once your Azure instance is configured (either manually or with PowerShell) to export flow logs to a storage account from the specified resource group and location, you'll return to Kentik to specify general settings for the cloud export:
- If you're on the Configure Flow Log Export step, click the Next button, which will take you to step 4, Name this Cloud Export.
- In the Name field, enter the name that should be shown for the cloud export in the Cloud Exports List on the Public Clouds page.
- In the Description field, briefly describe the Azure resources covered by this cloud export.
- From the Billing Plan drop-down, choose the Kentik billing plan to which the cloud export should be assigned (see About Plans).
- Click Next to go to the next step.
Validate Azure Setup
At this point you're nearly done with the setup of an Azure cloud export. To complete the process, the recommended procedure in most cases would be as follows:
- In the Validate Configuration pane, click the Validate button to begin validation of your flow log export configuration.
- When the two checkmarks below appear in the tile, click the Save button.
- Access to storage account
- Access to Azure Cloud metadata
- The Monitor your Azure Cloud wizard will close, and validation will continue in the background.
Note: Validation may take up to an hour, during which time the cloud export's status (e.g. on Public Clouds Page) will be indicated as "Pending" until Kentik completes registration.