Proposed Pull Request Change

📄 Document Links

View on GitHub View on Microsoft Learn

--- title: 'Azure HDInsight: Azure CLI samples' description: Azure CLI examples for common tasks in Azure HDInsight. ms.service: azure-hdinsight ms.topic: sample ms.date: 09/06/2024 ms.custom: devx-track-azurecli author: apurbasroy ms.author: apsinhar ms.reviewer: hgowrisankar --- # Azure HDInsight: Azure CLI samples > [!div class="op_single_selector"] > [.NET Examples](hdinsight-sdk-dotnet-samples.md) > [Python Examples](hdinsight-sdk-python-samples.md) > [Java Examples](hdinsight-sdk-java-samples.md) This article provides sample scripts for common tasks. For each example, update the variables with the appropriate values, and then execute the command. ## Prerequisites * Azure CLI. See [Install the Azure CLI](/cli/azure/install-azure-cli) for steps. * Optional: Bash. The examples in this article use the Bash shell on Windows 10. See [Windows Subsystem for Linux Installation Guide for Windows 10](/windows/wsl/install-win10) for installation steps. The examples work from a Windows Command prompt with some slight modifications. ## `az login` [Sign in to Azure](/cli/azure/reference-index#az-login). ```azurecli az login # If you have multiple subscriptions, set the one to use # az account set --subscription "SUBSCRIPTIONID" ``` ## `az hdinsight create` [Creates a new cluster](/cli/azure/hdinsight#az-hdinsight-create). ### Create a cluster with an existing storage account ```azurecli # set variables export clusterName=CLUSTERNAME export resourceGroupName=RESOURCEGROUPNAME export clusterType=hadoop export httpCredential='PASSWORD' export AZURE_STORAGE_ACCOUNT=STORAGEACCOUNTNAME az hdinsight create \ --name $clusterName \ --resource-group $resourceGroupName \ --type $clusterType \ --http-password $httpCredential \ --storage-account $AZURE_STORAGE_ACCOUNT ``` ### Create a cluster with the Enterprise Security Package (ESP) ```azurecli export clusterName=CLUSTERNAME export resourceGroupName=RESOURCEGROUPNAME export clusterType=spark export httpCredential='PASSWORD' export AZURE_STORAGE_ACCOUNT=STORAGEACCOUNTNAME export subnet="/subscriptions/aaaa0a0a-bb1b-cc2c-dd3d-eeeeee4e4e4e/resourceGroups/MyRG/providers/Microsoft.Network/virtualNetworks/MyVnet/subnets/subnet1" export domain="/subscriptions/aaaa0a0a-bb1b-cc2c-dd3d-eeeeee4e4e4e/resourceGroups/MyRG/providers/Microsoft.AAD/domainServices/MyDomain.onmicrosoft.com" export userAssignedIdentity="/subscriptions/aaaa0a0a-bb1b-cc2c-dd3d-eeeeee4e4e4e/resourcegroups/MyMsiRG/providers/Microsoft.ManagedIdentity/userAssignedIdentities/MyMSI" export domainAccount=MyAdminAccount@MyDomain.onmicrosoft.com export groupDNS=MyGroup az hdinsight create \ --esp \ --name $clusterName \ --resource-group $resourceGroupName \ --type $clusterType \ --http-password $httpCredential \ --storage-account $AZURE_STORAGE_ACCOUNT \ --subnet $subnet \ --domain $domain \ --assign-identity $userAssignedIdentity \ --cluster-admin-account $domainAccount \ --cluster-users-group-dns $groupDNS ``` ### Create a Kafka cluster with [disk encryption](./disk-encryption.md) ```azurecli export clusterName=CLUSTERNAME export resourceGroupName=RESOURCEGROUPNAME export clusterType=kafka export httpCredential='PASSWORD' export AZURE_STORAGE_ACCOUNT=STORAGEACCOUNTNAME export encryptionKeyName=kafkaClusterKey export encryptionKeyVersion=00000000000000000000000000000000 export encryptionVaultUri=https://MyKeyVault.vault.azure.net export userAssignedIdentity="/subscriptions/aaaa0a0a-bb1b-cc2c-dd3d-eeeeee4e4e4e/resourcegroups/MyMsiRG/providers/Microsoft.ManagedIdentity/userAssignedIdentities/MyMSI" az hdinsight create \ --name $clusterName \ --resource-group $resourceGroupName \ --type $clusterType \ --http-password $httpCredential \ --storage-account $AZURE_STORAGE_ACCOUNT \ --workernode-data-disks-per-node 2 \ --encryption-key-name $encryptionKeyName \ --encryption-key-version $encryptionKeyVersion \ --encryption-vault-uri $encryptionVaultUri \ --assign-identity $userAssignedIdentity ``` ### Create a cluster with Azure Data Lake Storage Gen2 ```azurecli export clusterName=CLUSTERNAME export resourceGroupName=RESOURCEGROUPNAME export clusterType=spark export httpCredential='PASSWORD' export adlgen2=MyStorageAccount export sami=MyMSI az hdinsight create \ --name $clusterName \ --resource-group $resourceGroupName \ --type $clusterType \ --http-password $httpCredential \ --storage-account $adlgen2 \ --storage-account-managed-identity $sami ``` ### Create a cluster with configuration from JSON string ```azurecli export clusterName=CLUSTERNAME export resourceGroupName=RESOURCEGROUPNAME export clusterType=spark export httpCredential='PASSWORD' export AZURE_STORAGE_ACCOUNT=STORAGEACCOUNTNAME export clusterConfiguration="{'gateway':{'restAuthCredential.username':'admin'}}" az hdinsight create \ --name $clusterName \ --resource-group $resourceGroupName \ --type $clusterType \ --http-password $httpCredential \ --storage-account $AZURE_STORAGE_ACCOUNT \ --cluster-configuration $clusterConfiguration ``` ### Create a cluster with configuration from a local file ```azurecli export clusterName=CLUSTERNAME export resourceGroupName=RESOURCEGROUPNAME export clusterType=spark export httpCredential='PASSWORD' export AZURE_STORAGE_ACCOUNT=STORAGEACCOUNTNAME export clusterConfiguration=@/mnt/c/HDI/config.json az hdinsight create \ --name $clusterName \ --resource-group $resourceGroupName \ --type $clusterType \ --http-password $httpCredential \ --storage-account $AZURE_STORAGE_ACCOUNT \ --cluster-configuration $clusterConfiguration ``` ## `az hdinsight application create` [Create an application for a HDInsight cluster](/cli/azure/hdinsight/application#az-hdinsight-application-create). ### Create an application with a script URI ```azurecli export resourceGroupName=RESOURCEGROUPNAME export applicationName=MyApplication export clusterName=CLUSTERNAME export scriptURI=https://hdiconfigactions.blob.core.windows.net/linuxhueconfigactionv02/install-hue-uber-v02.sh export scriptActionName=MyScriptAction export scriptParameters='"-version latest -port 20000"' az hdinsight application create \ --resource-group $resourceGroupName \ --name $applicationName \ --cluster-name $clusterName \ --script-uri $scriptURI \ --script-action-name $scriptActionName \ --script-parameters "$scriptParameters" ``` ### Create an application with a script URI and specified edge node size ```azurecli export resourceGroupName=RESOURCEGROUPNAME export applicationName=MyApplication export clusterName=CLUSTERNAME export scriptURI=https://hdiconfigactions.blob.core.windows.net/linuxhueconfigactionv02/install-hue-uber-v02.sh export scriptActionName=MyScriptAction export scriptParameters='"-version latest -port 20000"' export edgenodeSize=Standard_D4_v2 az hdinsight application create \ --resource-group $resourceGroupName \ --name $applicationName \ --cluster-name $clusterName \ --script-uri $scriptURI \ --script-action-name $scriptActionName \ --script-parameters "$scriptParameters" \ --edgenode-size $edgenodeSize ``` ### Create an application with HTTPS Endpoint ```azurecli export resourceGroupName=RESOURCEGROUPNAME export applicationName=MyApplication export clusterName=CLUSTERNAME export scriptURI=https://hdiconfigactions.blob.core.windows.net/linuxhueconfigactionv02/install-hue-uber-v02.sh export scriptActionName=MyScriptAction export scriptParameters='"-version latest -port 20000"' export destinationPort=8888 export subDomainSuffix=was az hdinsight application create \ --resource-group $resourceGroupName \ --name $applicationName \ --cluster-name $clusterName \ --script-uri $scriptURI \ --script-action-name $scriptActionName \ --script-parameters "$scriptParameters" \ --destination-port $destinationPort \ --sub-domain-suffix $subDomainSuffix ``` ## `az hdinsight script-action execute` [Execute script actions on the specified HDInsight cluster](/cli/azure/hdinsight/script-action#az-hdinsight-script-action-execute). ### Execute a script action and persist on success ```azurecli export resourceGroupName=RESOURCEGROUPNAME export clusterName=CLUSTERNAME export scriptActionName=MyScriptAction export scriptURI=https://hdiconfigactions.blob.core.windows.net/linuxgiraphconfigactionv01/giraph-installer-v01.sh export roles="headnode workernode" az hdinsight script-action execute \ --resource-group $resourceGroupName \ --cluster-name $clusterName \ --name $scriptActionName \ --script-uri $scriptURI \ --roles $roles \ --persist-on-success ```