Kentik for Azure
The export of flow logs from Microsoft Azure, which enables a unified view of network traffic data in hybrid cloud environments, is covered in the following topics:
About Azure Flow Logs
The basics of Azure flow logs are covered in the following topics:
Azure Flow Log Overview
Microsoft Azure is a cloud computing service created by Microsoft for use alone or in a hybrid configuration in which Azure resources work with other cloud resources and/or with traditional data centers (as shown below). With the introduction of support for Azure flow logs, Kentik now enables network traffic data from Azure resources to be used alongside data from other supported sources — both physical and cloud — for visualization, monitoring, alerting, and analytics.
In Azure, flow logging is a service provided by Azure Monitor, which has the resource provider namespace Microsoft.Insights. Flow logs are generated by network security groups. Each NSG represents resources in your Azure subscription, such as virtual machines (VMs), on which network traffic comes (ingress) and goes (egress). The flow logs, structured as JSON, provide the flow data detailed in the Azure documentation topic Log file.
Kentik does not consume flow logs directly from the monitored resources. Instead, the log files are exported to a storage account that includes the logs generated by all NSGs that share a given location and have been assigned to the same resource group. The storage account is accessed by NSG Flow Exporter (a Kentik-built enterprise application for Azure), which forwards the flow logs to KDE (see About Kentik Data Engine). The KDE ingest layer then normalizes the logs to Kentik’s internal records format, enriches each record with Azure-specific details, and stores the resulting records in KDE for access via Kentik.
Azure Flow Log Formats
The JSON flow records created by Azure flow logging cover all inbound and outbound IP flows for each NSG rule and include the flow’s network interface (NIC), 5-tuple information, traffic decision, and throughput information (Version 2 only). For details, see the following Azure documentation topics:
Azure Flow Log Retention
Deletion of flow logs after they’ve been ingested by Kentik enables you to minimize the costs associated with log data retention in the cloud. In Azure, log deletion is controlled by a retention setting that determines how long after its creation each log is kept. The way the retention duration is set depends on the method you choose to configure log export (see Choose Configuration Method):
- If you configure via script (recommended), the default value for log retention will be two days. You can change this setting by modifying the script as described in step 4 of Generate PowerShell Script.
- If you configure manually, you can change the Retention setting in Azure’s Flow logs settings dialog, which you’ll be directed to by the instructions provided when you click the Configure Manually button in the Configure Flow Log Export tile of Kentik’s Add Azure Cloud dialog.
Note: In Azure, retention is specified as an integer representing whole days, with a valid range of 1 -365.
Azure Flow Log Resources
For further information about Azure, please refer to the following Microsoft Azure documentation:
Azure Logging Setup Overview
Azure logging begins with creating a Cloud (see About Clouds) in the Kentik portal. The workflow for creating and configuring the Cloud is as follows:
- In your Azure portal, gather the information that you’ll need later in Kentik (see Gather Azure Information).
- In the Add Azure Cloud dialog (Kentik portal), complete general setting for the Cloud (Azure Cloud General Settings).
- In the Add Azure Cloud dialog, authorize Kentik’s Azure application, NSG Flow Exporter, to access the Azure directory containing the resources from which you want to export flow logs to Kentik (Authorize Access to Azure).
- In the Add Azure Cloud dialog, specify the Azure resources from which you want to export logs (Specify Azure Resources).
- In the Add Azure Cloud dialog, generate a script based on the information provided in earlier steps, then use that script in the Azure portal to configure Azure export settings (Configure Azure Log Export).
- Back in the Add Azure Cloud dialog, initiate validation of the configuration and complete registration of the Cloud (Validate Azure Setup).
Note: The settings and controls of the Add Azure Cloud dialog in the Kentik portal are listed in Azure Provider Settings.
Azure Clouds in the Portal
Successful completion of the tasks listed in the workflow above will have the following effect in the Kentik portal:
- A new Cloud will be shown as an added row in the Kentik portal’s Clouds list (Admin » Clouds; see Clouds List). The Cloud will represent the collection of Azure resources whose logs are pulled by Kentik from one Azure storage account.
- The Devices column in the Clouds list will show one cloud device, the name of which will match the name you give to the storage account.
- The Device Groups List on the Cloud Details page for the new device will list one device group, the name of which will match the name you give to the storage account.
Gather Azure Information
Adding an Azure Cloud in Kentik involves an exchange of information between Kentik and the Azure portal. To complete the setup process, you’ll need to gather information from your account in the Azure portal (https://portal.azure.com) as covered in the following topics:
Check Azure Role
To add an Azure Cloud to Kentik your role in Azure must allow you to grant the permissions that enable Kentik’s NSG Flow Exporter application to access the resources (e.g. your “default directory”) from which you’ll be sending flow logs to Kentik. Kentik has confirmed that the role of “Global Administrator” works for this purpose.
Note: To see if other roles in your organization (e.g. Application Administrator or Cloud Application Administrator) may also be able to grant the required permissions please refer to the Azure document Available roles.
To confirm that you are a global administrator or other qualifying role:
- On the Home page of your Azure portal, click Azure Active Directory in the sidebar at left.
- In the resulting Default Directory - Overview page, find the Manage list (second sidebar from the left) and click Roles and administrators.
- Your role will be indicated at the top of the main page (above the Administrative roles heading).
If you’re not a Global Administrator (or other qualifying role), you’ll need to be made one by an administrator of your Azure account. If you are one, you may wish to designate other Azure users so that they can also add Azure Clouds in Kentik. Either way, the process is as follows:
- In the Administrative roles list (main display area) of the Roles and administrators page, click Global Administrator.
- On the resulting Global administrator - Members page, you’ll see a list of users who are global administrators. Above the list, click Add member.
Note: If you’re not a global administrator this option will be grayed out.
- In the resulting Add Member popup you’ll see a list of members. Find the user(s) that you want to designate as a global administrator (use the Select field to filter the list) and click them to add them to the Selected members list.
- Click the Select button at the bottom of the popup. The popup will close and the selected members will now be present in the list of global administrators. Anyone in that list who is also registered as a user in Kentik should be able to add an Azure Cloud.
Note: For further information on the assignment of administrator roles, please refer to the Azure documentation View and assign roles.
Find Azure Subscription ID
Assets in Azure are each associated with a subscription. To add a Cloud, you’ll need to find the subscription ID for the Azure directory containing the assets from which you want to export flow logs:
- On the Home page of your Azure portal, click All Services in the sidebar at left.
- At the left of the main All Services page, click General to filter the list of services.
- In the list of General services, click on Subscriptions.
- A table at the bottom of the resulting Subscriptions page will list all of your subscriptions. Copy and save the 32-digit GUID in the Subscription ID row of the subscription associated with the assets from which you want to send flow logs to Kentik.
Find Resource Group and Location
A Cloud in Kentik represents a set of assets in Azure that share a unique Storage Account. All assets in the storage account must be in the same Resource Group and Location. To create a Storage Account you’ll need to find the Resource Group and Location for the assets:
- On the Home page of your Azure portal, click Virtual Machines in the sidebar at left.
- In the table on the resulting page, find one of the VMs from which you want to export flow logs.
- Copy and save the values in the Resource Group and Location columns.
Azure Cloud General Settings
Once you have the required information from the Azure portal, you’re ready to start adding an Azure Cloud in Kentik. Begin by opening the Add Azure Cloud dialog and specifying general settings for the Cloud (see Common Cloud Settings):
- Click Admin on the main portal navbar.
- At the top of the resulting the Admin page you’ll see the Add Data Sources pane. In the Microsoft Azure tile, click the Add button to open the Add Azure Cloud dialog (see Cloud Admin Dialogs).
- Enter the Name and Description for the Cloud.
- Assign the Cloud to one of your Kentik billing plans (see About Plans).
- Set the Enabled switch to On.
Authorize Access to Azure
To export flow logs from assets in Azure, you’ll need to grant permission for NSG Flow Exporter to access the Azure directory containing those assets. You’ll do this by authorizing Azure to create a “service principal” representing NSG Flow Exporter:
- In the Azure Provider Settings pane of the Add Azure Cloud dialog, you’ll see a Subscription ID field (shown at right) in the Authorize Access to Azure tile. Enter the ID of the subscription associated with the Azure directory containing the assets (see Find Azure Subscription ID).
- In the same tile, click the Authorize button, which will take you to the Azure portal.
- On the Azure login screen, pick the account associated with the assets from which you want to send flow logs to Kentik.
- Once you’re logged in, you’ll be taken to the Permissions requested screen (shown at right), where you’ll be asked to grant the permissions required for NSG Flow Exporter to access resources in your account.
Note: If you are not taken to the Permissions requested screen, and are instead returned to the Kentik portal immediately after login, check that your Azure role allows you to grant the needed permissions (see Check Azure Role).
- Click the Accept button, which will create an Azure service principal for the NSG Flow Exporter application. You will then be returned to the Add Azure Cloud dialog in the Kentik portal.
Specify Azure Resources
Flow logging requires two types of resources in Azure:
- The Azure resources (e.g. VM) for which you are generating logs.
- A “storage account” where the logs are collected and from which they are forwarded to Kentik (specifically to the Kentik Data Engine).
To set up log export, these resources must be identified to Kentik in the Azure Provider Settings tile of the Add Azure Cloud dialog:
- In the Resource Group to Monitor field, enter the resource group that you gathered in Find Resource Group and Location.
- In the Azure Location field, enter the location that you gathered in Find Resource Group and Location.
- In the Storage Account Name field, enter a name for the storage account to which logs will be exported from the above-specified Azure resources.
Note: The name must not be already in use by any other storage account, whether in your subscription or that of another Azure user.
Configure Azure Log Export
The next step is to configure your Azure instance to export flow logs, which is covered in the following topics:
Choose Configuration Method
Next you’ll configure your Azure instance to export flow logs from the specified resource group and location to an Azure storage account. You can choose either of the following two methods in the Configure Flow Log Export tile of the Add Azure Cloud dialog:
- Configure via script (recommended): When you click the Configure Using PowerShell button, Kentik will generate a script based on the information provided in Specify Azure Resources. You will then run the script in the Azure portal’s PowerShell instance as described in Configure Using PowerShell.
- Configure manually: Clicking the Configure Manually button will open a dialog containing a set of steps. Go to the Azure portal and follow the instructions.
Note: To get a feel for the process involved in manual configuration, see the Microsoft Azure document Tutorial: Log network traffic.
Generate PowerShell Script
To generate the PowerShell script:
- In the Configure Flow Log Export tile, click the Configure Using PowerShell button, which opens the Logging Configuration Script dialog. The dialog includes a Kentik-generated script.
- Check that the values for resource group, location, and storage account that are listed near the top of the dialog are correct. If not, close the dialog and correct the information that you entered in the Azure Provider Settings tile.
- Click the Copy to Clipboard button, then close the dialog.
- If you’d like the logs to be retained for a duration other than the default two days:
- Paste the script into a text editor.
- Modify the value of the RetentionInDays argument in the declaration of the $ret variable (currently on line 167). The new value must be an integer, representing whole days, between 1 and 365.
- Copy the edited script back to the clipboard.
Configure Using PowerShell
To configure Azure log export in the Azure portal using PowerShell:
- Navigate to your Azure portal and log in.
- In the main Azure navbar, click the PowerShell icon (>_). PowerShell will open and initialize.
- When initialization is complete, type code at the prompt. The code editor will open.
- Paste the script from Generate PowerShell Script into the code editor, then save the script:
- Using the drop-down menu (“...” icon) at the upper right of the script editor, open the Save new file dialog.
- Enter a name for the script. The name must end with.ps1 extension, e.g. MyKentikScript.ps1.
- Click the Save button.
- Choose Close from the script editor menu to close the editor.
- Back in PowerShell, enter the full path to the script at the prompt, e.g.:
Note: The user_name for an Azure script file path is the first word of the full user name, e.g. if your user name is “Sallie Mae” then sallie.
- The script will run (see Azure Script Operations). When it’s done you’ll see a confirming message with the information that you entered in the Add Azure Cloud dialog: subscription ID, resource group, location, and storage account.
Note: PowerShell instances are ephemeral. If you are timed out or otherwise lose connection during the procedure above, you must re-start the process from step 2.
Azure Script Operations
The configuration script executes a series of operations that configure Azure for log export based on your settings in the Add Azure Cloud dialog. These operations include the following:
- Confirm existence of the provided subscription ID.
- Confirm existence of the specified location.
- Confirm existence of the specified resource group.
- Confirm that a service principal has been created in your Azure instance for the Kentik application NSG Flow Exporter.
- Confirm that NSG Flow Exporter has been granted the required access (“Reader”) to the specified resource group.
- Create a storage account with the specified name for the specified resource group and location.
- Grant NSG Flow Exporter the required access (“Contributor”) to the storage account.
- Confirm that the Network Watcher feature is registered in your subscription.
- Confirm that Network Watcher exists for the specified resource group and location.
- Confirm that the specified subscription is registered with Microsoft.Insights, the resource provider namespace for Azure Monitor, which provides the logging resource.
- Build a list of network security groups (NSGs) found in the specified resource group and location.
- Enable v2 NSG flow logs for each found NSG.
Validate Azure Setup
Once you’ve successfully run the configuration script in the Azure PowerShell you are nearly done with Azure Cloud setup. To complete the process, the recommended procedure in most cases would be as follows:
- In the Validate Configuration tile, click the Validate button to begin validation of your flow log export configuration.
- Click the Add Azure Cloud button after the two checkmarks below appear in the tile.
- Access to storage account
- Access to Azure Cloud metadata
- The dialog will close, and validation will continue in the background.
Alternatively, there may be situations in which you choose not to validate and instead close the Add Azure Cloud dialog immediately. This would allow you to, for example, pre-provision an Azure Cloud in Kentik before setting up flow export manually in Azure.
Note: Validation may take up to an hour, during which time the Cloud’s status (e.g. on Clouds Page) will be indicated as “Pending” until Kentik completes registration.