Proposed Pull Request Change

📄 Document Links

View on GitHub View on Microsoft Learn

--- title: Automatic update of the Mobility service in Azure Site Recovery description: Overview of automatic update of the Mobility service when replicating Azure VMs by using Azure Site Recovery. author: Jeronika-MS ms.service: azure-site-recovery ms.topic: how-to ms.date: 03/24/2023 ms.author: v-gajeronika ms.custom: - engagement-fy23 - sfi-image-nochange # Customer intent: As an IT administrator managing Azure virtual machines, I want to automate the updates of the Mobility service through Site Recovery, so that I can ensure efficient disaster recovery without manually overseeing each upgrade. --- # Automatic update of the Mobility service in Azure-to-Azure replication Azure Site Recovery uses a monthly release cadence to fix any issues and enhance existing features or add new ones. To remain current with the service, you must plan for patch deployment each month. To avoid the overhead associated with each upgrade, you can allow Site Recovery to manage component updates. As mentioned in [Azure-to-Azure disaster recovery architecture](azure-to-azure-architecture.md), the Mobility service is installed on all Azure virtual machines (VMs) that have replication enabled from one Azure region to another. When you use automatic updates, each new release updates the Mobility service extension. [!INCLUDE [updated-for-az](~/reusable-content/ce-skilling/azure/includes/updated-for-az.md)] ## How automatic updates work When you use Site Recovery to manage updates, it deploys a global runbook (used by Azure services) via an automation account, created in the same subscription as the vault. Each vault uses one automation account. For each VM in a vault, the runbook checks for active auto-updates. If a newer version of the Mobility service extension is available, the update is installed. The default runbook schedule occurs daily at 12:00 AM in the time zone of the replicated VM's geography. You can also change the runbook schedule via the automation account. > [!NOTE] > Starting with [Update Rollup 35](site-recovery-whats-new-archive.md#updates-march-2019), you can choose an existing automation account to use for updates. Prior to Update Rollup 35, Site Recovery created the automation account by default. You can only select this option when you enable replication for a VM. It isn't available for a VM that already has replication enabled. The setting you select applies to all Azure VMs protected in the same vault. Turning on automatic updates doesn't require a restart of your Azure VMs or affect ongoing replication. Job billing in the automation account is based on the number of job runtime minutes used in a month. Job execution takes a few seconds to about a minute each day and is covered as free units. By default, 500 minutes are included as free units for an automation account, as shown in the following table: | Free units included (each month) | Price | |---|---| | Job runtime 500 minutes | ₹0.14/minute ## Enable automatic updates There are several ways that Site Recovery can manage the extension updates: - [Manage as part of the enable replication step](#manage-as-part-of-the-enable-replication-step) - [Toggle the extension update settings inside the vault](#toggle-the-extension-update-settings-inside-the-vault) - [Manage updates manually](#manage-updates-manually) ### Manage as part of the enable replication step When you enable replication for a VM either starting [from the VM view](azure-to-azure-quickstart.md) or [from the recovery services vault](azure-to-azure-how-to-enable-replication.md), you can either allow Site Recovery to manage updates for the Site Recovery extension or manage it manually. :::image type="content" source="./media/azure-to-azure-autoupdate/enable-rep.png" alt-text="Extension settings"::: ### Toggle the extension update settings inside the vault 1. From the Recovery Services vault, go to **Manage** > **Site Recovery Infrastructure**. 1. Under **For Azure Virtual Machines** > **Extension Update Settings** > **Allow Site Recovery to manage**, select **On**. To manage the extension manually, select **Off**. > [!IMPORTANT] > When you choose **Allow Site Recovery to manage**, the setting is applied to all VMs in the vault. 1. Select **Save**. :::image type="content" source="./media/azure-to-azure-autoupdate/vault-toggle.png" alt-text="Extension update settings"::: > [!NOTE] > Either option notifies you of the automation account used for managing updates. If you're using this feature in a vault for the first time, a new automation account is created by default. Alternately, you can customize the setting, and choose an existing automation account. Once defined, all subsequent actions to enable replication in the same vault will use that selected automation account. Currently, the drop-down menu will only list automation accounts that are in the same Resource Group as the vault. **For a custom automation account, use the following script:** > [!IMPORTANT] > Run the following script in the context of an automation account. This script leverages System Assigned Managed Identities as its authentication type. ```azurepowershell param( [Parameter(Mandatory=$true)] [String] $VaultResourceId, [Parameter(Mandatory=$true)] [ValidateSet("Enabled",'Disabled')] [Alias("Enabled or Disabled")] [String] $AutoUpdateAction, [Parameter(Mandatory=$false)] [String] $AutomationAccountArmId ) $SiteRecoveryRunbookName = "Modify-AutoUpdateForVaultForPartner" $TaskId = [guid]::NewGuid().ToString() $SubscriptionId = "aaaa0a0a-bb1b-cc2c-dd3d-eeeeee4e4e4e" $AsrApiVersion = "2021-12-01" $ArmEndPoint = "https://management.azure.com" $AadAuthority = "https://login.windows.net/" $AadAudience = "https://management.core.windows.net/" $AzureEnvironment = "AzureCloud" $Timeout = "160" $AuthenticationType = "SystemAssignedIdentity" function Throw-TerminatingErrorMessage { Param ( [Parameter(Mandatory=$true)] [String] $Message ) throw ("Message: {0}, TaskId: {1}.") -f $Message, $TaskId } function Write-Tracing { Param ( [Parameter(Mandatory=$true)] [ValidateSet("Informational", "Warning", "ErrorLevel", "Succeeded", IgnoreCase = $true)] [String] $Level, [Parameter(Mandatory=$true)] [String] $Message, [Switch] $DisplayMessageToUser ) Write-Output $Message } function Write-InformationTracing { Param ( [Parameter(Mandatory=$true)] [String] $Message ) Write-Tracing -Message $Message -Level Informational -DisplayMessageToUser } function ValidateInput() { try { if(!$VaultResourceId.StartsWith("/subscriptions", [System.StringComparison]::OrdinalIgnoreCase)) { $ErrorMessage = "The vault resource id should start with /subscriptions." throw $ErrorMessage } $Tokens = $VaultResourceId.SubString(1).Split("/") if(!($Tokens.Count % 2 -eq 0)) { $ErrorMessage = ("Odd Number of tokens: {0}." -f $Tokens.Count) throw $ErrorMessage } if(!($Tokens.Count/2 -eq 4)) { $ErrorMessage = ("Invalid number of resource in vault ARM id expected:4, actual:{0}." -f ($Tokens.Count/2)) throw $ErrorMessage } if($AutoUpdateAction -ieq "Enabled" -and [string]::IsNullOrEmpty($AutomationAccountArmId)) { $ErrorMessage = ("The automation account ARM id should not be null or empty when AutoUpdateAction is enabled.") throw $ErrorMessage } } catch { $ErrorMessage = ("ValidateInput failed with [Exception: {0}]." -f $_.Exception) Write-Tracing -Level ErrorLevel -Message $ErrorMessage -DisplayMessageToUser Throw-TerminatingErrorMessage -Message $ErrorMessage } } function Initialize-SubscriptionId() { try { $Tokens = $VaultResourceId.SubString(1).Split("/") $Count = 0 $ArmResources = @{} while($Count -lt $Tokens.Count) { $ArmResources[$Tokens[$Count]] = $Tokens[$Count+1] $Count = $Count + 2 } return $ArmResources["subscriptions"] } catch { Write-Tracing -Level ErrorLevel -Message ("Initialize-SubscriptionId: failed with [Exception: {0}]." -f $_.Exception) -DisplayMessageToUser throw } } function Invoke-InternalRestMethod($Uri, $Headers, [ref]$Result) { $RetryCount = 0 $MaxRetry = 3 do { try { $ResultObject = Invoke-RestMethod -Uri $Uri -Headers $Headers ($Result.Value) += ($ResultObject) break } catch { Write-InformationTracing ("Retry Count: {0}, Exception: {1}." -f $RetryCount, $_.Exception) $RetryCount++ if(!($RetryCount -le $MaxRetry)) { throw } Start-Sleep -Milliseconds 2000 } }while($true) } function Invoke-InternalWebRequest($Uri, $Headers, $Method, $Body, $ContentType, [ref]$Result) { $RetryCount = 0 $MaxRetry = 3 do { try { $ResultObject = Invoke-WebRequest -Uri $UpdateUrl -Headers $Header -Method 'PATCH' ` -Body $InputJson -ContentType "application/json" -UseBasicParsing ($Result.Value) += ($ResultObject) break } catch { Write-InformationTracing ("Retry Count: {0}, Exception: {1}." -f $RetryCount, $_.Exception) $RetryCount++ if(!($RetryCount -le $MaxRetry)) { throw } Start-Sleep -Milliseconds 2000 } }while($true) } function Get-Header([ref]$Header, $AadAudience){ try { $Header.Value['Content-Type'] = 'application\json' Write-InformationTracing ("The Authentication Type is system Assigned Identity based.") $endpoint = $env:IDENTITY_ENDPOINT $endpoint $Headers = New-Object "System.Collections.Generic.Dictionary[[String],[String]]" $Headers.Add("X-IDENTITY-HEADER", $env:IDENTITY_HEADER) $Headers.Add("Metadata", "True") $authenticationResult = Invoke-RestMethod -Method Get -Headers $Headers -Uri ($endpoint +'?resource=' +$AadAudience) $accessToken = $authenticationResult.access_token $Header.Value['Authorization'] = "Bearer " + $accessToken $Header.Value["x-ms-client-request-id"] = $TaskId + "/" + (New-Guid).ToString() + "-" + (Get-Date).ToString("u") } catch { $ErrorMessage = ("Get-BearerToken: failed with [Exception: {0}]." -f $_.Exception) Write-Tracing -Level ErrorLevel -Message $ErrorMessage -DisplayMessageToUser Throw-TerminatingErrorMessage -Message $ErrorMessage } } function Get-ProtectionContainerToBeModified([ref] $ContainerMappingList) { try { Write-InformationTracing ("Get protection container mappings : {0}." -f $VaultResourceId) $ContainerMappingListUrl = $ArmEndPoint + $VaultResourceId + "/replicationProtectionContainerMappings" + "?api-version=" + $AsrApiVersion Write-InformationTracing ("Getting the bearer token and the header.") Get-Header ([ref]$Header) $AadAudience $Result = @() Invoke-InternalRestMethod -Uri $ContainerMappingListUrl -Headers $header -Result ([ref]$Result) $ContainerMappings = $Result[0] Write-InformationTracing ("Total retrieved container mappings: {0}." -f $ContainerMappings.Value.Count) foreach($Mapping in $ContainerMappings.Value) { if(($Mapping.properties.providerSpecificDetails -eq $null) -or ($Mapping.properties.providerSpecificDetails.instanceType -ine "A2A")) { Write-InformationTracing ("Mapping properties: {0}." -f ($Mapping.properties)) Write-InformationTracing ("Ignoring container mapping: {0} as the provider does not match." -f ($Mapping.Id)) continue; } if($Mapping.Properties.State -ine "Paired") { Write-InformationTracing ("Ignoring container mapping: {0} as the state is not paired." -f ($Mapping.Id)) continue; } Write-InformationTracing ("Provider specific details {0}." -f ($Mapping.properties.providerSpecificDetails)) $MappingAutoUpdateStatus = $Mapping.properties.providerSpecificDetails.agentAutoUpdateStatus $MappingAutomationAccountArmId = $Mapping.properties.providerSpecificDetails.automationAccountArmId $MappingHealthErrorCount = $Mapping.properties.HealthErrorDetails.Count if($AutoUpdateAction -ieq "Enabled" -and ($MappingAutoUpdateStatus -ieq "Enabled") -and ($MappingAutomationAccountArmId -ieq $AutomationAccountArmId) -and ($MappingHealthErrorCount -eq 0)) { Write-InformationTracing ("Provider specific details {0}." -f ($Mapping.properties)) Write-InformationTracing ("Ignoring container mapping: {0} as the auto update is already enabled and is healthy." -f ($Mapping.Id)) continue; } ($ContainerMappingList.Value).Add($Mapping.id) } } catch { $ErrorMessage = ("Get-ProtectionContainerToBeModified: failed with [Exception: {0}]." -f $_.Exception) Write-Tracing -Level ErrorLevel -Message $ErrorMessage -DisplayMessageToUser Throw-TerminatingErrorMessage -Message $ErrorMessage } } $OperationStartTime = Get-Date $ContainerMappingList = New-Object System.Collections.Generic.List[System.String] $JobsInProgressList = @() $JobsCompletedSuccessList = @() $JobsCompletedFailedList = @() $JobsFailedToStart = 0 $JobsTimedOut = 0 $Header = @{} $AzureRMProfile = Get-Module -ListAvailable -Name AzureRM.Profile | Select Name, Version, Path $AzureRmProfileModulePath = Split-Path -Parent $AzureRMProfile.Path Add-Type -Path (Join-Path $AzureRmProfileModulePath "Microsoft.IdentityModel.Clients.ActiveDirectory.dll") $Inputs = ("Tracing inputs VaultResourceId: {0}, Timeout: {1}, AutoUpdateAction: {2}, AutomationAccountArmId: {3}." -f $VaultResourceId, $Timeout, $AutoUpdateAction, $AutomationAccountArmId) Write-Tracing -Message $Inputs -Level Informational -DisplayMessageToUser $CloudConfig = ("Tracing cloud configuration ArmEndPoint: {0}, AadAuthority: {1}, AadAudience: {2}." -f $ArmEndPoint, $AadAuthority, $AadAudience) Write-Tracing -Message $CloudConfig -Level Informational -DisplayMessageToUser ValidateInput $SubscriptionId = Initialize-SubscriptionId Get-ProtectionContainerToBeModified ([ref]$ContainerMappingList) $Input = @{ "properties"= @{ "providerSpecificInput"= @{ "instanceType" = "A2A" "agentAutoUpdateStatus" = $AutoUpdateAction "automationAccountArmId" = $AutomationAccountArmId "automationAccountAuthenticationType" = $AuthenticationType } } } $InputJson = $Input | ConvertTo-Json if ($ContainerMappingList.Count -eq 0) { Write-Tracing -Level Succeeded -Message ("Exiting as there are no container mappings to be modified.") -DisplayMessageToUser exit } Write-InformationTracing ("Container mappings to be updated has been retrieved with count: {0}." -f $ContainerMappingList.Count) try { Write-InformationTracing ("Start the modify container mapping jobs.") ForEach($Mapping in $ContainerMappingList) { try { $UpdateUrl = $ArmEndPoint + $Mapping + "?api-version=" + $AsrApiVersion Get-Header ([ref]$Header) $AadAudience $Result = @() Invoke-InternalWebRequest -Uri $UpdateUrl -Headers $Header -Method 'PATCH' ` -Body $InputJson -ContentType "application/json" -Result ([ref]$Result) $Result = $Result[0] $JobAsyncUrl = $Result.Headers['Azure-AsyncOperation'] Write-InformationTracing ("The modify container mapping job invoked with async url: {0}." -f $JobAsyncUrl) $JobsInProgressList += $JobAsyncUrl; # Rate controlling the set calls to maximum 60 calls per minute. # ASR throttling for set calls is 200 in 1 minute. Start-Sleep -Milliseconds 1000 } catch{ Write-InformationTracing ("The modify container mappings job creation failed for: {0}." -f $Ru) Write-InformationTracing $_ $JobsFailedToStart++ } } Write-InformationTracing ("Total modify container mappings has been initiated: {0}." -f $JobsInProgressList.Count) } catch { $ErrorMessage = ("Modify container mapping jobs failed with [Exception: {0}]." -f $_.Exception) Write-Tracing -Level ErrorLevel -Message $ErrorMessage -DisplayMessageToUser Throw-TerminatingErrorMessage -Message $ErrorMessage } try { while($JobsInProgressList.Count -ne 0) { Sleep -Seconds 30 $JobsInProgressListInternal = @() ForEach($JobAsyncUrl in $JobsInProgressList) { try { Get-Header ([ref]$Header) $AadAudience $Result = Invoke-RestMethod -Uri $JobAsyncUrl -Headers $header $JobState = $Result.Status if($JobState -ieq "InProgress") { $JobsInProgressListInternal += $JobAsyncUrl } elseif($JobState -ieq "Succeeded" -or ` $JobState -ieq "PartiallySucceeded" -or ` $JobState -ieq "CompletedWithInformation") { Write-InformationTracing ("Jobs succeeded with state: {0}." -f $JobState) $JobsCompletedSuccessList += $JobAsyncUrl } else { Write-InformationTracing ("Jobs failed with state: {0}." -f $JobState) $JobsCompletedFailedList += $JobAsyncUrl } } catch { Write-InformationTracing ("The get job failed with: {0}. Ignoring the exception and retrying the next job." -f $_.Exception) # The job on which the tracking failed, will be considered in progress and tried again later. $JobsInProgressListInternal += $JobAsyncUrl } # Rate controlling the get calls to maximum 120 calls each minute. # ASR throttling for get calls is 10000 in 60 minutes. Start-Sleep -Milliseconds 500 } Write-InformationTracing ("Jobs remaining {0}." -f $JobsInProgressListInternal.Count) $CurrentTime = Get-Date if($CurrentTime -gt $OperationStartTime.AddMinutes($Timeout)) { Write-InformationTracing ("Tracing modify cloud pairing jobs has timed out.") $JobsTimedOut = $JobsInProgressListInternal.Count $JobsInProgressListInternal = @() } $JobsInProgressList = $JobsInProgressListInternal } } catch { $ErrorMessage = ("Tracking modify cloud pairing jobs failed with [Exception: {0}]." -f $_.Exception) Write-Tracing -Level ErrorLevel -Message $ErrorMessage -DisplayMessageToUser Throw-TerminatingErrorMessage -Message $ErrorMessage } Write-InformationTracing ("Tracking modify cloud pairing jobs completed.") Write-InformationTracing ("Modify cloud pairing jobs success: {0}." -f $JobsCompletedSuccessList.Count) Write-InformationTracing ("Modify cloud pairing jobs failed: {0}." -f $JobsCompletedFailedList.Count) Write-InformationTracing ("Modify cloud pairing jobs failed to start: {0}." -f $JobsFailedToStart) Write-InformationTracing ("Modify cloud pairing jobs timedout: {0}." -f $JobsTimedOut) if($JobsTimedOut -gt 0) { $ErrorMessage = "One or more modify cloud pairing jobs has timedout." Write-Tracing -Level ErrorLevel -Message ($ErrorMessage) Throw-TerminatingErrorMessage -Message $ErrorMessage } elseif($JobsCompletedSuccessList.Count -ne $ContainerMappingList.Count) { $ErrorMessage = "One or more modify cloud pairing jobs failed." Write-Tracing -Level ErrorLevel -Message ($ErrorMessage) Throw-TerminatingErrorMessage -Message $ErrorMessage } Write-Tracing -Level Succeeded -Message ("Modify cloud pairing completed.") -DisplayMessageToUser ``` ### Manage updates manually 1. If there are new updates for the Mobility service installed on your VMs, you'll see the following notification: **New Site Recovery replication agent update is available. Click to install.** :::image type="content" source="./media/vmware-azure-install-mobility-service/replicated-item-notif.png" alt-text="Replicated items window"::: 1. Select the notification to open the VM selection page. 1. Choose the VMs you want to upgrade, and then select **OK**. The Update Mobility service will start for each selected VM. :::image type="content" source="./media/vmware-azure-install-mobility-service/update-okpng.png" alt-text="Replicated items VM list"::: ## Common issues and troubleshooting If there's an issue with the automatic updates, you'll see an error notification under **Configuration issues** in the vault dashboard. If you can't enable automatic updates, see the following common errors and recommended actions: - **Error**: You do not have permissions to create an Azure Run As account (service principal) and grant the Contributor role to the service principal. **Recommended action**: Make sure that the signed-in account is assigned as Contributor and try again. For more information about assigning permissions, see the required permissions section of [How to: Use the portal to create a Microsoft Entra application and service principal that can access resources](../active-directory/develop/howto-create-service-principal-portal.md#permissions-required-for-registering-an-app). To fix most issues after you enable automatic updates, select **Repair**. If the repair button isn't available, see the error message displayed in the extension update settings pane. :::image type="content" source="./media/azure-to-azure-autoupdate/repair.png" alt-text="Site Recovery service repair button in extension update settings"::: - **Error**: The Run As account does not have the permission to access the recovery services resource. **Recommended action**: Delete and then [re-create the Run As account](../automation/manage-runas-account.md). Or, make sure that the Automation Run As account's Microsoft Entra application can access the recovery services resource. - **Error**: Run As account is not found. Either one of these was deleted or not created - Microsoft Entra Application, Service Principal, Role, Automation Certificate asset, Automation Connection asset - or the Thumbprint is not identical between Certificate and Connection. **Recommended action**: Delete and then [re-create the Run As account](../automation/manage-runas-account.md). - **Error**: The Azure Run as Certificate used by the automation account is about to expire. The self-signed certificate that is created for the Run As account expires one year from the date of creation. You can renew it at any time before it expires. If you have signed up for email notifications, you will also receive emails when an action is required from your side. This error will be shown two months prior to the expiry date, and will change to a critical error if the certificate has expired. Once the certificate has expired, auto update will not be functional until you renew the same. **Recommended action**: To resolve this issue, select **Repair** and then **Renew Certificate**. > [!NOTE] > After you renew the certificate, refresh the page to display the current status. ## Next steps [Learn more](./how-to-migrate-run-as-accounts-managed-identity.md) on how to migrate the authentication type of the Automation accounts to Managed Identities.