Document Details
Hdinsight Hadoop Customize Cluster Bootstrap
❌
This page contains Windows bias
About This Page
This page is part of the Azure documentation. It contains code examples and configuration instructions for working with Azure services.
Bias Analysis
Bias Types:
⚠️
powershell_heavy
⚠️
windows_first
⚠️
missing_linux_example
⚠️
windows_tools
Summary:
The documentation page is heavily biased toward Windows and PowerShell usage. All code samples and detailed walkthroughs are provided exclusively in PowerShell, with no equivalent Bash, CLI, or Linux-native scripting examples. The prerequisites and appendix focus solely on Windows tooling (PowerShell, .NET SDK), and the only mention of Linux is a link to a separate page, rather than integrated examples. There is no demonstration of how to perform the same tasks using Azure CLI, Bash scripts, or Linux-native tools.
Recommendations:
- Add equivalent examples using Azure CLI and Bash scripts for customizing HDInsight clusters, demonstrating parity with the PowerShell examples.
- Include Linux prerequisites (e.g., Azure CLI installation) alongside PowerShell/Az Module instructions.
- Present cross-platform options (Azure CLI, REST API, ARM templates) before or alongside Windows-specific tools like PowerShell.
- Integrate Linux-focused examples directly into the main documentation, rather than relegating them to separate pages.
- Explicitly mention that all approaches are cross-platform where applicable, and clarify any Windows-only limitations.
Create pull request
Scan History
| Date | Scan ID | Status | Bias Status |
|---|---|---|---|
| 2025-09-16 00:00 | #113 | completed | ✅ Clean |
| 2025-09-15 00:00 | #112 | completed | ✅ Clean |
| 2025-09-14 00:00 | #111 | completed | ✅ Clean |
| 2025-09-13 00:00 | #110 | completed | ✅ Clean |
| 2025-09-12 00:00 | #109 | completed | ✅ Clean |
| 2025-09-11 00:00 | #108 | completed | ✅ Clean |
| 2025-09-10 00:00 | #107 | completed | ✅ Clean |
| 2025-09-09 00:00 | #106 | completed | ✅ Clean |
| 2025-09-08 00:00 | #105 | completed | ✅ Clean |
| 2025-09-07 00:00 | #104 | completed | ✅ Clean |
| 2025-09-06 00:00 | #103 | completed | ✅ Clean |
| 2025-08-17 00:01 | #83 | in_progress | ✅ Clean |
| 2025-07-13 21:37 | #48 | completed | ✅ Clean |
| 2025-07-12 23:44 | #41 | in_progress | ❌ Biased |
| 2025-07-09 13:09 | #3 | cancelled | ✅ Clean |
| 2025-07-08 04:23 | #2 | cancelled | ❌ Biased |
Flagged Code Snippets
# hdfs-site.xml configuration
$HdfsConfigValues = @{ "dfs.blocksize"="64m" } #default is 128MB in HDI 3.0 and 256MB in HDI 2.1
# core-site.xml configuration
$CoreConfigValues = @{ "ipc.client.connect.max.retries"="60" } #default 50
# mapred-site.xml configuration
$MapRedConfigValues = @{ "mapreduce.task.timeout"="1200000" } #default 600000
# oozie-site.xml configuration
$OozieConfigValues = @{ "oozie.service.coord.normal.default.timeout"="150" } # default 120
# hive-site.xml configuration
$hiveConfigValues = @{ "hive.metastore.client.socket.timeout"="90s" }
$config = New-AzHDInsightClusterConfig `
-ClusterType "Spark" `
| Set-AzHDInsightDefaultStorage `
-StorageAccountResourceId "$storageAccountResourceId" `
-StorageAccountKey $defaultStorageAccountKey `
| Add-AzHDInsightConfigValue `
-HiveSite $hiveConfigValues `
-Spark2Defaults @{}
New-AzHDInsightCluster `
-ResourceGroupName $resourceGroupName `
-ClusterName $hdinsightClusterName `
-Location $location `
-ClusterSizeInNodes 2 `
-Version "4.0" `
-HttpCredential $httpCredential `
-SshCredential $sshCredential `
-Config $config
####################################
# Service names and variables
####################################
$nameToken = "<ENTER AN ALIAS>"
$namePrefix = $nameToken.ToLower() + (Get-Date -Format "MMdd")
$resourceGroupName = $namePrefix + "rg"
$hdinsightClusterName = $namePrefix + "hdi"
$defaultStorageAccountName = $namePrefix + "store"
$defaultBlobContainerName = $hdinsightClusterName
$location = "East US"
####################################
# Connect to Azure
####################################
Write-Host "Connecting to your Azure subscription ..." -ForegroundColor Green
$sub = Get-AzSubscription -ErrorAction SilentlyContinue
if(-not($sub))
{
Connect-AzAccount
}
# If you have multiple subscriptions, set the one to use
#$context = Get-AzSubscription -SubscriptionId "<subscriptionID>"
#Set-AzContext $context
####################################
# Create a resource group
####################################
Write-Host "Creating a resource group ..." -ForegroundColor Green
New-AzResourceGroup `
-Name $resourceGroupName `
-Location $location
####################################
# Create a storage account and container
####################################
Write-Host "Creating the default storage account and default blob container ..." -ForegroundColor Green
New-AzStorageAccount `
-ResourceGroupName $resourceGroupName `
-Name $defaultStorageAccountName `
-Location $location `
-SkuName Standard_LRS `
-Kind StorageV2 `
-EnableHttpsTrafficOnly 1
$defaultStorageAccountKey = (Get-AzStorageAccountKey `
-ResourceGroupName $resourceGroupName `
-Name $defaultStorageAccountName)[0].Value
$defaultStorageContext = New-AzStorageContext `
-StorageAccountName $defaultStorageAccountName `
-StorageAccountKey $defaultStorageAccountKey
New-AzStorageContainer `
-Name $defaultBlobContainerName `
-Context $defaultStorageContext #use the cluster name as the container name
####################################
# Create a configuration object
####################################
$hiveConfigValues = @{"hive.metastore.client.socket.timeout"="90s"}
$storageAccountResourceId = (Get-AzStorageAccount -ResourceGroupName $resourceGroupName ` -Name $defaultStorageAccountName).Id
$config = New-AzHDInsightClusterConfig `
-ClusterType "Spark" `
| Set-AzHDInsightDefaultStorage `
-StorageAccountResourceId "$storageAccountResourceId" `
-StorageAccountKey $defaultStorageAccountKey `
| Add-AzHDInsightConfigValue `
-HiveSite $hiveConfigValues `
-Spark2Defaults @{}
####################################
# Set Ambari admin username/password
####################################
$httpUserName = "admin" #HDInsight cluster username
$httpPassword = '<ENTER A PASSWORD>'
$httpPW = ConvertTo-SecureString -String $httpPassword -AsPlainText -Force
$httpCredential = New-Object System.Management.Automation.PSCredential($httpUserName,$httpPW)
####################################
# Set ssh username/password
####################################
$sshUserName = "sshuser" #HDInsight ssh user name
$sshPassword = '<ENTER A PASSWORD>'
$sshPW = ConvertTo-SecureString -String $sshPassword -AsPlainText -Force
$sshCredential = New-Object System.Management.Automation.PSCredential($sshUserName,$sshPW)
####################################
# Create an HDInsight cluster
####################################
New-AzHDInsightCluster `
-ResourceGroupName $resourceGroupName `
-ClusterName $hdinsightClusterName `
-Location $location `
-ClusterSizeInNodes 2 `
-Version "4.0" `
-HttpCredential $httpCredential `
-SshCredential $sshCredential `
-Config $config
####################################
# Verify the cluster
####################################
Get-AzHDInsightCluster `
-ClusterName $hdinsightClusterName