Flow/Firewall Log Collection

This article covers how to set up Azure flow and firewall logs in Kentik.

Azure Flow Log Collection

Kentik supports Azure flow logs, allowing network traffic data from Azure to be integrated with other sources for visualization, monitoring, alerting, and analytics in the Kentik portal.

Key Points:

• Flow Logging Service:

  • Provided by Azure Monitor under the namespace Microsoft.Insights.

  • Flow logs are generated by VNets or NSGs in your Azure subscription.

  • Logs capture ingress and egress traffic and are structured in JSON format (see Azure docs on Log format).

• Log Export Process:

  • Logs are not consumed directly from resources. Instead, they are exported to a storage account that aggregates logs from NSGs/VNets in the same location and resource group.

NSG Flow Logs vs. VNet Flow Logs

Azure supports the collection of both VNet flow logs and NSG flow logs, as compared here:

• NSG Flow Logs: Capture IP traffic flowing through an NSG.

  • NSGs are used to filter network traffic to and from Azure resources in a virtual network.

  • NSG flow logs provide detailed information about each flow, including source and destination IP addresses, ports, protocol, and whether the traffic was allowed or denied by the NSG rule.

  • Offer granular visibility at the NSG level, useful for understanding security group efficacy and troubleshooting access issues for specific subnets or VMs.

• VNet Flow Logs: Capture traffic entering and leaving the VNet.

  • Ideal when managing complex network topologies with many NSGs.

  • Simplify troubleshooting and security monitoring for the entire virtual network.

TIP: Kentik recommends following Microsoft’s guidance to use VNet flow logs instead of NSG flow logs going forward (see Microsoft's flow logging recommendation).

Azure Flow Log Retention

Managing the retention of Azure flow logs is crucial for minimizing cloud storage costs. Here’s how retention is handled:

• Retention Setting: Determines how long each log is kept after creation. It can be configured in two ways:

  • Via Script (Recommended):

    • Default retention is set to two days.

    • Modify the retention duration in the script as described in step #4 of Generate PowerShell Script.

  • Manual Configuration: Adjust the Retention setting in Azure’s Flow logs settings dialog.

• Retention Range:

  • Specified as an integer in whole days.

  • Valid range is from 1 to 365 days.

Enable VNet/NSG Flow Log Export

To enable VNet or NSG flow log export to a storage account, follow these steps:

1. Enable the Network Watcher service in the region where your resources reside:

   1. In the Azure Portal, search for and select Network Watcher.

   2. Ensure the status for your target region is Enabled.

2. Configure the flow logs:

   1. In the Network Watcher menu, select Flow logs.

   2. Click + Create to set up a new flow log.

   3. For Target Resource, select the VNet or NSG you wish to monitor.

   4. Select the Storage account you will use for Kentik.

   5. Finalize the setting to begin the export.

Azure Firewall Log Collection

In addition to VNet and NSG flow logs, Kentik supports the ingestion of Azure Firewall logs to provide deeper visibility into application and network-level security events.

Notes:

• Firewall logs must be sent to the same storage account used for your VNet flow logs.

• In addition to the Azure docs linked below, see the process steps in Enable Firewall Log Export.

Supported Firewall Log Types

Kentik can process the following Azure Firewall log types (click a type to view the related Azure doc):

• azfwapplicationrule: Logs for application rules associated with a firewall.

• azfwnetworkrule: Logs for network rules associated with a firewall.

• Legacy Azure diagnostics logs: Includes support for AzureFirewallNetworkRule and AzureFirewallApplicationRule.

Enable Firewall Log Export

To enable Azure Firewall log export to the same storage account used for your VNet/NSG flow logs, follow these steps:

1. Create a diagnostic setting for one or more log categories:

   1. In the Azure Portal, navigate to your Azure Firewall resource.

   2. In the left-hand menu under Monitoring, select Diagnostic settings.

   3. Click + Add diagnostic setting.

   4. In the Log categories section, select one or more of the following:

      • azfwapplicationrule: For application-level filtering logs.

      • azfwnetworkrule: For network-level filtering logs.

      • AzureFirewallNetworkRule / AzureFirewallApplicationRule: If using legacy log formats.

   5. Under Destination details, check the box for Archive to a storage account.

   6. Select the Subscription and the specific Storage account that Kentik is configured to monitor.