Proposed Pull Request Change

📄 Document Links

View on GitHub View on Microsoft Learn

--- title: Use Azure Monitor logs to monitor Azure HDInsight clusters description: Learn how to use Azure Monitor logs to monitor jobs running in a HDInsight cluster. ms.service: azure-hdinsight ms.topic: how-to ms.custom: devx-track-azurepowershell, references_regions, devx-track-azurecli author: yeturis ms.author: sairamyeturi ms.reviewer: nijelsf ms.date: 10/15/2024 ROBOTS: NOINDEX --- # Use Azure Monitor logs to monitor HDInsight clusters Learn how to enable Azure Monitor logs to monitor Hadoop cluster operations in HDInsight. And how to add a HDInsight monitoring solution. [Azure Monitor logs](/azure/azure-monitor/logs/log-query-overview) is an Azure Monitor service that monitors your cloud and on-premises environments. The monitoring is to maintain their availability and performance. It collects data generated by resources in your cloud, on-premises environments and from other monitoring tools. The data is used to provide analysis across multiple sources. [!INCLUDE [azure-monitor-log-analytics-rebrand](~/reusable-content/ce-skilling/azure/includes/azure-monitor-log-analytics-rebrand.md)] If you don't have an Azure subscription, [create a free account](https://azure.microsoft.com/pricing/purchase-options/azure-account?cid=msft_learn) before you begin. #### [New Azure monitor experience](#tab/new) > [!Important] > Azure Monitor experience (preview) in HDInsight is retiring by February 1, 2025. For more information, see [Retirement: Azure Monitor experience (preview) in HDInsight is retiring by February 1, 2025](https://azure.microsoft.com/updates/v2/HDInsight-Azure-Monitor-experience-retirement). ## Prerequisites * A Log Analytics workspace. You can think of this workspace as a unique Azure Monitor logs environment with its own data repository, data sources, and solutions. For the instructions, see [Create a Log Analytics workspace](/azure/azure-monitor/logs/quick-create-workspace). * An Azure HDInsight cluster. Currently, you can use Azure Monitor logs with the following HDInsight cluster types: * Hadoop * HBase * Interactive Query * Kafka * Spark For the instructions on how to create a HDInsight cluster, see [Get started with Azure HDInsight](hadoop/apache-hadoop-linux-tutorial-get-started.md). * If using PowerShell, you need the [Az Module](/powershell/azure/). Ensure you have the latest version. If necessary, run `Update-Module -Name Az`. * If wanting to use Azure CLI and you haven't yet installed it, see [Install the Azure CLI](/cli/azure/install-azure-cli). > [!NOTE] > New Azure Monitor experience is only available in all the regions as a preview feature. It is recommended to place both the HDInsight cluster and the Log Analytics workspace in the same region for better performance. > ## Enable Azure Monitor using the portal In this section, you configure an existing HDInsight Hadoop cluster to use an Azure Log Analytics workspace to monitor jobs, debug logs, and so on. 1. From the [Azure portal](https://portal.azure.com/), select your cluster. The cluster is opened in a new portal page. 2. From the left, under Monitoring, select **Monitor Integration**. 3. From the main view, under **Azure Monitor for HDInsight Clusters Integration**, select **Enable**. 4. From the **Select a workspace** drop-down list, select an existing Log Analytics workspace. 5. Select **Save**. It takes a few moments to save the setting. :::image type="content" source="./media/hdinsight-hadoop-oms-log-analytics-tutorial/hdinsight-enable-azure-monitor.png" alt-text="Enable monitoring for HDInsight clusters."::: If you want to disable Azure Monitor, you can do the same in this portal. ## Enable Azure Monitor using Azure PowerShell You can enable Azure Monitor logs using the Azure PowerShell Az module [Enable-AzHDInsightAzureMonitor](/powershell/module/az.hdinsight/enable-azhdinsightazuremonitor) cmdlet. ```powershell # Enter user information $resourceGroup = "<your-resource-group>" $cluster = "<your-cluster>" $LAW = "<your-Log-Analytics-workspace>" # End of user input # obtain workspace id for defined Log Analytics workspace $WorkspaceId = (Get-AzOperationalInsightsWorkspace ` -ResourceGroupName $resourceGroup ` -Name $LAW).CustomerId # obtain primary key for defined Log Analytics workspace $PrimaryKey = (Get-AzOperationalInsightsWorkspace ` -ResourceGroupName $resourceGroup ` -Name $LAW | Get-AzOperationalInsightsWorkspaceSharedKeys).PrimarySharedKey # Enables monitoring and relevant logs will be sent to the specified workspace. Enable-AzHDInsightAzureMonitor ` -ResourceGroupName $resourceGroup ` -ClusterName $cluster ` -WorkspaceId $WorkspaceId ` -PrimaryKey $PrimaryKey # Gets the status of monitoring installation on the cluster. Get-AzHDInsightAzureMonitor ` -ResourceGroupName $resourceGroup ` -ClusterName $cluster ``` To disable, the use the [Disable-AzHDInsightAzureMonitor](/powershell/module/az.hdinsight/disable-azhdinsightazuremonitor) cmdlet: ```powershell Disable-AzHDInsightAzureMonitor -ResourceGroupName $resourceGroup ` -ClusterName $cluster ``` ## Enable Azure Monitor using Azure CLI You can enable Azure Monitor logs using the Azure CLI [`az hdinsight azure-monitor enable`](/cli/azure/hdinsight/azure-monitor) command. ```azurecli # set variables export resourceGroup=RESOURCEGROUPNAME export cluster=CLUSTERNAME export LAW=LOGANALYTICSWORKSPACENAME # Enable the Azure Monitor logs integration on an HDInsight cluster. az hdinsight azure-monitor enable --name $cluster --resource-group $resourceGroup --workspace $LAW # Get the status of Azure Monitor logs integration on an HDInsight cluster. az hdinsight azure-monitor show --name $cluster --resource-group $resourceGroup ``` To disable, the use the [`az hdinsight monitor disable`](/cli/azure/hdinsight/monitor#az-hdinsight-monitor-disable) command. ```azurecli az hdinsight azure-monitor disable --name $cluster --resource-group $resourceGroup ``` ## Use HDInsight out-of-box Insights to monitor a single cluster HDInsight provides workload-specific workbook to help you quickly get insights. This workbook collects important performance metrics from your HDInsight cluster and provides the visualizations and dashboards for most common scenarios. The out-of-box insights give a complete view of a single HDInsight cluster including resource utilization and application status. Available HDInsight workbooks: - HDInsight Spark Workbook - HDInsight Kafka Workbook - HDInsight HBase Workbook - HDInsight Hive/LLAP Workbook Screenshot of Spark Workbook :::image type="content" source="./media/hdinsight-hadoop-oms-log-analytics-tutorial/hdinsight-spark-workbook.png" alt-text="Spark workbook screenshot."::: ## Configuring performance counters Azure monitor supports collecting and analyzing performance metrics for the nodes in your cluster. For more information, see [Linux performance data sources in Azure Monitor](/azure/azure-monitor/agents/data-sources-performance-counters#linux-performance-counters). ## Cluster auditing HDInsight support cluster auditing with Azure Monitor logs, by importing the following types of logs: * `log_gateway_audit_CL` - this table provides audit logs from cluster gateway nodes that show successful and failed sign-in attempts. * `log_auth_CL` - this table provides SSH logs with successful and failed sign-in attempts. * `log_ambari_audit_CL` - this table provides audit logs from Ambari. * `ranger_audit_logs_CL` - this table provides audit logs from Apache Ranger on ESP clusters. For the log table mappings from the classic Azure Monitor integration to the new one, see [Log table mapping](monitor-hdinsight-reference.md#log-table-mapping). #### [Classic Azure Monitor experience](#tab/previous) > [!Important] > On 31 August 2024, Azure is retiring the Classic Azure Monitor experience on HDInsight. ## Prerequisites * A Log Analytics workspace. You can think of this workspace as a unique Azure Monitor logs environment with its own data repository, data sources, and solutions. For the instructions, see [Create a Log Analytics workspace](/azure/azure-monitor/vm/monitor-virtual-machine). * If you intend to use Azure Monitor integration on a cluster behind a firewall, complete the [Prerequisites for clusters behind a firewall](#oms-with-firewall). * An Azure HDInsight cluster. Currently, you can use Azure Monitor logs with the following HDInsight cluster types: * Hadoop * HBase * Interactive Query * Kafka * Spark For the instructions on how to create a HDInsight cluster, see [Get started with Azure HDInsight](hadoop/apache-hadoop-linux-tutorial-get-started.md). * If using PowerShell, you'll need the [Az Module](/powershell/azure/). Ensure you have the latest version. If necessary, run `Update-Module -Name Az`. * If wanting to use Azure CLI and you haven't yet installed it, see [Install the Azure CLI](/cli/azure/install-azure-cli). > [!NOTE] > It is recommended to place both the HDInsight cluster and the Log Analytics workspace in the same region for better performance. Azure Monitor logs is not available in all Azure regions. ## Enable Azure Monitor using the portal In this section, you configure an existing HDInsight Hadoop cluster to use an Azure Log Analytics workspace to monitor jobs, debug logs, and so on. 1. From the [Azure portal](https://portal.azure.com/), select your cluster. The cluster is opened in a new portal page. 1. From the left, under **Monitoring**, select **Azure Monitor**. 1. From the main view, under **Azure Monitor Integration**, select **Enable**. 1. From the **Select a workspace** drop-down list, select an existing Log Analytics workspace. 1. Select **Save**. It takes a few moments to save the setting. :::image type="content" source="./media/hdinsight-hadoop-oms-log-analytics-tutorial/azure-portal-monitoring.png" alt-text="Enable monitoring for HDInsight clusters."::: ## Enable Azure Monitor using Azure PowerShell You can enable Azure Monitor logs using the Azure PowerShell Az module [Enable-AzHDInsightMonitoring](/powershell/module/az.hdinsight/enable-azhdinsightmonitoring) cmdlet. ```powershell # Enter user information $resourceGroup = "<your-resource-group>" $cluster = "<your-cluster>" $LAW = "<your-Log-Analytics-workspace>" # End of user input # obtain workspace id for defined Log Analytics workspace $WorkspaceId = (Get-AzOperationalInsightsWorkspace ` -ResourceGroupName $resourceGroup ` -Name $LAW).ResourceId # obtain primary key for defined Log Analytics workspace $PrimaryKey = (Get-AzOperationalInsightsWorkspace ` -ResourceGroupName $resourceGroup ` -Name $LAW | Get-AzOperationalInsightsWorkspaceSharedKeys).PrimarySharedKey # Enables monitoring and relevant logs will be sent to the specified workspace. Enable-AzHDInsightMonitoring ` -ResourceGroupName $resourceGroup ` -Name $cluster ` -WorkspaceId $WorkspaceId ` -PrimaryKey $PrimaryKey # Gets the status of monitoring installation on the cluster. Get-AzHDInsightMonitoring ` -ResourceGroupName $resourceGroup ` -Name $cluster ``` To disable, the use the [Disable-AzHDInsightMonitoring](/powershell/module/az.hdinsight/disable-azhdinsightmonitoring) cmdlet: ```powershell Disable-AzHDInsightMonitoring -Name "<your-cluster>" ``` ## Enable Azure Monitor using Azure CLI You can enable Azure Monitor logs using the Azure CLI `[az hdinsight monitor enable`](/cli/azure/hdinsight/monitor#az-hdinsight-monitor-enable) command. ```azurecli # set variables export resourceGroup=RESOURCEGROUPNAME export cluster=CLUSTERNAME export LAW=LOGANALYTICSWORKSPACENAME # Enable the Azure Monitor logs integration on an HDInsight cluster. az hdinsight monitor enable --name $cluster --resource-group $resourceGroup --workspace $LAW # Get the status of Azure Monitor logs integration on an HDInsight cluster. az hdinsight monitor show --name $cluster --resource-group $resourceGroup ``` To disable, the use the [`az hdinsight monitor disable`](/cli/azure/hdinsight/monitor#az-hdinsight-monitor-disable) command. ```azurecli az hdinsight monitor disable --name $cluster --resource-group $resourceGroup ``` ## <a name="oms-with-firewall"></a>Prerequisites for clusters behind a firewall To be able to successfully set up Azure Monitor integration with HDInsight, behind a firewall, some customers may need to enable the following endpoints: |Agent Resource | Ports | Direction | Bypass HTTPS inspection | |---|---|---|---| | \*.ods.opinsights.azure.com | Port 443 | Outbound | Yes | | \*.oms.opinsights.azure.com |Port 443 | Outbound | Yes | | \*.azure-automation.net | Port 443 | Outbound | Yes | If you have security restrictions related to enabling wildcard storage endpoints, there is an alternate option. You can do the following instead: 1. Create a dedicated storage account 2. Configure the dedicated storage account on their log analytics workspace 3. Enable that dedicated storage account in their firewall ### Data collection behind a firewall Once the setup is successful, enabling necessary endpoints for data ingestion is important. It is recommended that you enable the \*.blob.core.windows.net endpoint for data ingestion to succeed. ## Install HDInsight cluster management solutions HDInsight provides cluster-specific management solutions that you can add for Azure Monitor Logs. [Management solutions](/previous-versions/azure/azure-monitor/insights/solutions) add functionality to Azure Monitor Logs, providing more data and analysis tools. These solutions collect important performance metrics from your HDInsight clusters. And provide the tools to search the metrics. These solutions also provide visualizations and dashboards for most cluster types supported in HDInsight. By using the metrics that you collect with the solution, you can create custom monitoring rules and alerts. Available HDInsight solutions: * HDInsight Hadoop Monitoring * HDInsight HBase Monitoring * HDInsight Interactive Query Monitoring * HDInsight Kafka Monitoring * HDInsight Spark Monitoring For management solution instructions, see [Management solutions in Azure](/previous-versions/azure/azure-monitor/insights/solutions#install-a-monitoring-solution). To experiment, install a HDInsight Hadoop Monitoring solution. When it's done, you see an **HDInsightHadoop** tile listed under **Summary**. Select the **HDInsightHadoop** tile. The HDInsightHadoop solution looks like: :::image type="content" source="media/hdinsight-hadoop-oms-log-analytics-tutorial/hdinsight-oms-hdinsight-hadoop-monitoring-solution.png" alt-text="HDInsight monitoring solution view."::: Because the cluster is a brand new cluster, the report doesn't show any activities. ## Cluster auditing HDInsight support cluster auditing with Azure Monitor logs, by importing the following types of logs: * `log_gateway_audit_CL` - this table provides audit logs from cluster gateway nodes that show successful and failed sign-in attempts. * `log_auth_CL` - this table provides SSH logs with successful and failed sign-in attempts. * `log_ambari_audit_CL` - this table provides audit logs from Ambari. * `ranger_audit_logs_CL` - this table provides audit logs from Apache Ranger on ESP clusters. For the log table mappings from the classic Azure Monitor integration to the new one, see [Log table mapping](monitor-hdinsight-reference.md#log-table-mapping). --- ## Update the Log Analytics (OMS) Agent used by HDInsight Azure Monitor Integration When Azure Monitor integration is enabled on a cluster, the Log Analytics agent, or Operations Management Suite (OMS) Agent, is installed on the cluster and is not updated unless you disable and re-enable Azure Monitor Integration. Complete the following steps if you need to update the OMS Agent on the cluster. If you are behind a firewall you may need to complete the [Prerequisites for clusters behind a firewall](hdinsight-hadoop-oms-log-analytics-tutorial.md?tabs=previous#oms-with-firewall) before completing these steps. 1. From the [Azure portal](https://portal.azure.com/), select your cluster. The cluster is opened in a new portal page. 1. From the left, under **Monitoring**, select **Azure Monitor**. 1. Note the name of your current Log Analytics workspace. 1. From the main view, under **Azure Monitor Integration**, disable the toggle, and then select **Save**. 1. After the setting saves, re-enable the **Azure Monitor Integration** toggle, and ensure the same Log Analytics workspace is selected, and then select **Save**. If you have Azure Monitor Integration enabled on a cluster, updating the OMS agent will also update the Open Management Infrastructure (OMI) version. You can check the OMI version on the cluster by running the following command: ``` sudo /opt/omi/bin/omiserver –version ``` ## Next steps * [Selective logging analysis](selective-logging-analysis.md) * [Query Azure Monitor logs to monitor HDInsight clusters](hdinsight-hadoop-oms-log-analytics-use-queries.md) * [How to monitor cluster availability with Apache Ambari and Azure Monitor logs](./hdinsight-cluster-availability.md)