Proposed Pull Request Change

📄 Document Links

View on GitHub View on Microsoft Learn

--- title: Networking patterns for Azure Service Fabric description: Describes common networking patterns for Service Fabric and how to create a cluster by using Azure networking features. ms.topic: concept-article ms.author: tomcassidy author: tomvcassidy ms.service: azure-service-fabric services: service-fabric ms.date: 11/21/2024 ms.update-cycle: 1095-days # Customer intent: "As a cloud engineer, I want to configure a Service Fabric cluster with various Azure networking features, so that I can optimize network performance and ensure proper connectivity for my applications." --- # Service Fabric networking patterns You can integrate your Azure Service Fabric cluster with other Azure networking features. In this article, we show you how to create clusters that use the following features: - [Existing virtual network or subnet](#existingvnet) - [Static public IP address](#staticpublicip) - [Internal-only load balancer](#internallb) - [Internal and external load balancer](#internalexternallb) Service Fabric runs in a standard virtual machine scale set. Any functionality that you can use in a virtual machine scale set, you can use with a Service Fabric cluster. The networking sections of the Azure Resource Manager templates for virtual machine scale sets and Service Fabric are identical. After you deploy to an existing virtual network, it's easy to incorporate other networking features, like Azure ExpressRoute, Azure VPN Gateway, a network security group, and virtual network peering. ### Allowing the Service Fabric resource provider to query your cluster Service Fabric is unique from other networking features in one aspect. The [Azure portal](https://portal.azure.com) internally uses the Service Fabric resource provider to call to a cluster to get information about nodes and applications. The Service Fabric resource provider requires publicly accessible inbound access to the HTTP gateway port (port 19080, by default) on the management endpoint. [Service Fabric Explorer](service-fabric-visualizing-your-cluster.md) uses the management endpoint to manage your cluster. The Service Fabric resource provider also uses this port to query information about your cluster, to display in the Azure portal. If port 19080 is not accessible from the Service Fabric resource provider, a message like *Nodes Not Found* appears in the portal, and your node and application list appears empty. If you want to see your cluster in the Azure portal, your load balancer must expose a public IP address, and your network security group must allow incoming port 19080 traffic. If your setup does not meet these requirements, the Azure portal does not display the status of your cluster. [!INCLUDE [updated-for-az](~/reusable-content/ce-skilling/azure/includes/updated-for-az.md)] ## Templates All Service Fabric templates are in [GitHub](https://github.com/Azure/service-fabric-scripts-and-templates/tree/master/templates/networking). You should be able to deploy the templates as-is by using the following PowerShell commands. If you are deploying the existing Azure Virtual Network template or the static public IP template, first read the [Initial setup](#initialsetup) section of this article. <a id="initialsetup"></a> ## Initial setup ### Existing virtual network In the following example, we start with an existing virtual network named ExistingRG-vnet, in the **ExistingRG** resource group. The subnet is named default. These default resources are created when you use the Azure portal to create a standard virtual machine (VM). You could create the virtual network and subnet without creating the VM, but the main goal of adding a cluster to an existing virtual network is to provide network connectivity to other VMs. Creating the VM gives a good example of how an existing virtual network typically is used. If your Service Fabric cluster uses only an internal load balancer, without a public IP address, you can use the VM and its public IP as a secure *jump box*. ### Static public IP address A static public IP address generally is a dedicated resource that's managed separately from the VM or VMs it's assigned to. It's provisioned in a dedicated networking resource group (as opposed to in the Service Fabric cluster resource group itself). Create a static public IP address named staticIP1 in the same ExistingRG resource group, either in the Azure portal or by using PowerShell: ```powershell PS C:\Users\user> New-AzPublicIpAddress -Name staticIP1 -ResourceGroupName ExistingRG -Location westus -AllocationMethod Static -DomainNameLabel sfnetworking Name : staticIP1 ResourceGroupName : ExistingRG Location : westus Id : /subscriptions/1237f4d2-3dce-1236-ad95-123f764e7123/resourceGroups/ExistingRG/providers/Microsoft.Network/publicIPAddresses/staticIP1 Etag : W/"fc8b0c77-1f84-455d-9930-0404ebba1b64" ResourceGuid : 77c26c06-c0ae-496c-9231-b1a114e08824 ProvisioningState : Succeeded Tags : PublicIpAllocationMethod : Static IpAddress : 40.83.182.110 PublicIpAddressVersion : IPv4 IdleTimeoutInMinutes : 4 IpConfiguration : null DnsSettings : { "DomainNameLabel": "sfnetworking", "Fqdn": "sfnetworking.westus.cloudapp.azure.com" } ``` ### Service Fabric template In the examples in this article, we use the Service Fabric template.json. You can use the standard portal wizard to download the template from the portal before you create a cluster. You also can use one of the [sample templates](https://github.com/Azure-Samples/service-fabric-cluster-templates), like the [secure five-node Service Fabric cluster](https://github.com/Azure-Samples/service-fabric-cluster-templates/tree/master/5-VM-Windows-1-NodeTypes-Secure). <a id="existingvnet"></a> ## Existing virtual network or subnet 1. Change the subnet parameter to the name of the existing subnet, and then add two new parameters to reference the existing virtual network: ```json "subnet0Name": { "type": "string", "defaultValue": "default" }, "existingVNetRGName": { "type": "string", "defaultValue": "ExistingRG" }, "existingVNetName": { "type": "string", "defaultValue": "ExistingRG-vnet" }, /* "subnet0Name": { "type": "string", "defaultValue": "Subnet-0" }, "subnet0Prefix": { "type": "string", "defaultValue": "10.0.0.0/24" },*/ ``` You also can comment out the parameter with the name "virtualNetworkName" so that it won't prompt you to enter the virtual network name twice in the cluster deployment blade in the Azure portal. 2. Comment out `nicPrefixOverride` attribute of `Microsoft.Compute/virtualMachineScaleSets`, because you are using existing subnet and you have disabled this variable in step 1. ```json /*"nicPrefixOverride": "[parameters('subnet0Prefix')]",*/ ``` 3. Change the `vnetID` variable to point to the existing virtual network: ```json /*old "vnetID": "[resourceId('Microsoft.Network/virtualNetworks',parameters('virtualNetworkName'))]",*/ "vnetID": "[concat('/subscriptions/', subscription().subscriptionId, '/resourceGroups/', parameters('existingVNetRGName'), '/providers/Microsoft.Network/virtualNetworks/', parameters('existingVNetName'))]", ``` 4. Remove `Microsoft.Network/virtualNetworks` from your resources, so Azure does not create a new virtual network: ```json /*{ "apiVersion": "[variables('vNetApiVersion')]", "type": "Microsoft.Network/virtualNetworks", "name": "[parameters('virtualNetworkName')]", "location": "[parameters('computeLocation')]", "properties": { "addressSpace": { "addressPrefixes": [ "[parameters('addressPrefix')]" ] }, "subnets": [ { "name": "[parameters('subnet0Name')]", "properties": { "addressPrefix": "[parameters('subnet0Prefix')]" } } ] }, "tags": { "resourceType": "Service Fabric", "clusterName": "[parameters('clusterName')]" } },*/ ``` 5. Comment out the virtual network from the `dependsOn` attribute of `Microsoft.Compute/virtualMachineScaleSets`, so you don't depend on creating a new virtual network: ```json "apiVersion": "[variables('vmssApiVersion')]", "type": "Microsoft.Computer/virtualMachineScaleSets", "name": "[parameters('vmNodeType0Name')]", "location": "[parameters('computeLocation')]", "dependsOn": [ /*"[concat('Microsoft.Network/virtualNetworks/', parameters('virtualNetworkName'))]", */ "[Concat('Microsoft.Storage/storageAccounts/', variables('uniqueStringArray0')[0])]", ``` 6. Deploy the template: ```powershell New-AzResourceGroup -Name sfnetworkingexistingvnet -Location westus New-AzResourceGroupDeployment -Name deployment -ResourceGroupName sfnetworkingexistingvnet -TemplateFile C:\SFSamples\Final\template\_existingvnet.json ``` After deployment, your virtual network should include the new scale set VMs. The virtual machine scale set node type should show the existing virtual network and subnet. You also can use Remote Desktop Protocol (RDP) to access the VM that was already in the virtual network, and to ping the new scale set VMs: ``` C:>\Users\users>ping 10.0.0.5 -n 1 C:>\Users\users>ping NOde1000000 -n 1 ``` For another example, see [one that is not specific to Service Fabric](https://github.com/gbowerman/azure-myriad/tree/main/existing-vnet). <a id="staticpublicip"></a> ## Static public IP address 1. Add parameters for the name of the existing static IP resource group, name, and fully qualified domain name (FQDN): ```json "existingStaticIPResourceGroup": { "type": "string" }, "existingStaticIPName": { "type": "string" }, "existingStaticIPDnsFQDN": { "type": "string" } ``` 2. Remove the `dnsName` parameter. (The static IP address already has one.) ```json /* "dnsName": { "type": "string" }, */ ``` 3. Add a variable to reference the existing static IP address: ```json "existingStaticIP": "[concat('/subscriptions/', subscription().subscriptionId, '/resourceGroups/', parameters('existingStaticIPResourceGroup'), '/providers/Microsoft.Network/publicIPAddresses/', parameters('existingStaticIPName'))]", ``` 4. Remove `Microsoft.Network/publicIPAddresses` from your resources, so Azure does not create a new IP address: ```json /* { "apiVersion": "[variables('publicIPApiVersion')]", "type": "Microsoft.Network/publicIPAddresses", "name": "[concat(parameters('lbIPName'),)'-', '0')]", "location": "[parameters('computeLocation')]", "properties": { "dnsSettings": { "domainNameLabel": "[parameters('dnsName')]" }, "publicIPAllocationMethod": "Dynamic" }, "tags": { "resourceType": "Service Fabric", "clusterName": "[parameters('clusterName')]" } }, */ ``` 5. Comment out the IP address from the `dependsOn` attribute of `Microsoft.Network/loadBalancers`, so you don't depend on creating a new IP address: ```json "apiVersion": "[variables('lbIPApiVersion')]", "type": "Microsoft.Network/loadBalancers", "name": "[concat('LB', '-', parameters('clusterName'), '-', parameters('vmNodeType0Name'))]", "location": "[parameters('computeLocation')]", /* "dependsOn": [ "[concat('Microsoft.Network/publicIPAddresses/', concat(parameters('lbIPName'), '-', '0'))]" ], */ "properties": { ``` 6. In the `Microsoft.Network/loadBalancers` resource, change the `publicIPAddress` element of `frontendIPConfigurations` to reference the existing static IP address instead of a newly created one: ```json "frontendIPConfigurations": [ { "name": "LoadBalancerIPConfig", "properties": { "publicIPAddress": { /*"id": "[resourceId('Microsoft.Network/publicIPAddresses',concat(parameters('lbIPName'),'-','0'))]"*/ "id": "[variables('existingStaticIP')]" } } } ], ``` 7. In the `Microsoft.ServiceFabric/clusters` resource, change `managementEndpoint` to the DNS FQDN of the static IP address. If you are using a secure cluster, make sure you change *http://* to *https://*. (Note that this step applies only to Service Fabric clusters. If you are using a virtual machine scale set, skip this step.) ```json "fabricSettings": [], /*"managementEndpoint": "[concat('http://',reference(concat(parameters('lbIPName'),'-','0')).dnsSettings.fqdn,':',parameters('nt0fabricHttpGatewayPort'))]",*/ "managementEndpoint": "[concat('http://',parameters('existingStaticIPDnsFQDN'),':',parameters('nt0fabricHttpGatewayPort'))]", ``` 8. Deploy the template: ```powershell New-AzResourceGroup -Name sfnetworkingstaticip -Location westus $staticip = Get-AzPublicIpAddress -Name staticIP1 -ResourceGroupName ExistingRG $staticip New-AzResourceGroupDeployment -Name deployment -ResourceGroupName sfnetworkingstaticip -TemplateFile C:\SFSamples\Final\template\_staticip.json -existingStaticIPResourceGroup $staticip.ResourceGroupName -existingStaticIPName $staticip.Name -existingStaticIPDnsFQDN $staticip.DnsSettings.Fqdn ``` After deployment, you can see that your load balancer is bound to the public static IP address from the other resource group. The Service Fabric client connection endpoint and [Service Fabric Explorer](service-fabric-visualizing-your-cluster.md) endpoint point to the DNS FQDN of the static IP address. <a id="internallb"></a> ## Internal-only load balancer This scenario replaces the external load balancer in the default Service Fabric template with an internal-only load balancer. See [earlier in the article](#allowing-the-service-fabric-resource-provider-to-query-your-cluster) for implications for the Azure portal and for the Service Fabric resource provider. 1. Remove the `dnsName` parameter. (It's not needed.) ```json /* "dnsName": { "type": "string" }, */ ``` 2. Optionally, if you use a static allocation method, you can add a static IP address parameter. If you use a dynamic allocation method, you do not need to do this step. ```json "internalLBAddress": { "type": "string", "defaultValue": "10.0.0.250" } ``` 3. Remove `Microsoft.Network/publicIPAddresses` from your resources, so Azure does not create a new IP address: ```json /* { "apiVersion": "[variables('publicIPApiVersion')]", "type": "Microsoft.Network/publicIPAddresses", "name": "[concat(parameters('lbIPName'),)'-', '0')]", "location": "[parameters('computeLocation')]", "properties": { "dnsSettings": { "domainNameLabel": "[parameters('dnsName')]" }, "publicIPAllocationMethod": "Dynamic" }, "tags": { "resourceType": "Service Fabric", "clusterName": "[parameters('clusterName')]" } }, */ ``` 4. Remove the IP address `dependsOn` attribute of `Microsoft.Network/loadBalancers`, so you don't depend on creating a new IP address. Add the virtual network `dependsOn` attribute because the load balancer now depends on the subnet from the virtual network: ```json "apiVersion": "[variables('lbApiVersion')]", "type": "Microsoft.Network/loadBalancers", "name": "[concat('LB','-', parameters('clusterName'),'-',parameters('vmNodeType0Name'))]", "location": "[parameters('computeLocation')]", "dependsOn": [ /*"[concat('Microsoft.Network/publicIPAddresses/',concat(parameters('lbIPName'),'-','0'))]"*/ "[concat('Microsoft.Network/virtualNetworks/',parameters('virtualNetworkName'))]" ], ``` 5. Change the load balancer's `frontendIPConfigurations` setting from using a `publicIPAddress`, to using a subnet and `privateIPAddress`. `privateIPAddress` uses a predefined static internal IP address. To use a dynamic IP address, remove the `privateIPAddress` element, and then change `privateIPAllocationMethod` to **Dynamic**. ```json "frontendIPConfigurations": [ { "name": "LoadBalancerIPConfig", "properties": { /* "publicIPAddress": { "id": "[resourceId('Microsoft.Network/publicIPAddresses',concat(parameters('lbIPName'),'-','0'))]" } */ "subnet" :{ "id": "[variables('subnet0Ref')]" }, "privateIPAddress": "[parameters('internalLBAddress')]", "privateIPAllocationMethod": "Static" } } ], ``` 6. In the `Microsoft.ServiceFabric/clusters` resource, change `managementEndpoint` to point to the internal load balancer address. If you use a secure cluster, make sure you change *http://* to *https://*. (Note that this step applies only to Service Fabric clusters. If you are using a virtual machine scale set, skip this step.) ```json "fabricSettings": [], /*"managementEndpoint": "[concat('http://',reference(concat(parameters('lbIPName'),'-','0')).dnsSettings.fqdn,':',parameters('nt0fabricHttpGatewayPort'))]",*/ "managementEndpoint": "[concat('http://',reference(variables('lbID0')).frontEndIPConfigurations[0].properties.privateIPAddress,':',parameters('nt0fabricHttpGatewayPort'))]", ``` 7. Deploy the template: ```powershell New-AzResourceGroup -Name sfnetworkinginternallb -Location westus New-AzResourceGroupDeployment -Name deployment -ResourceGroupName sfnetworkinginternallb -TemplateFile C:\SFSamples\Final\template\_internalonlyLB.json ``` After deployment, your load balancer uses the private static 10.0.0.250 IP address. If you have another machine in that same virtual network, you can go to the internal [Service Fabric Explorer](service-fabric-visualizing-your-cluster.md) endpoint. Note that it connects to one of the nodes behind the load balancer. <a id="internalexternallb"></a> ## Internal and external load balancer In this scenario, you start with the existing single-node type external load balancer, and add an internal load balancer for the same node type. A back-end port attached to a back-end address pool can be assigned only to a single load balancer. Choose which load balancer will have your application ports, and which load balancer will have your management endpoints (ports 19000 and 19080). If you put the management endpoints on the internal load balancer, keep in mind the Service Fabric resource provider restrictions discussed [earlier in the article](#allowing-the-service-fabric-resource-provider-to-query-your-cluster). In the example we use, the management endpoints remain on the external load balancer. You also add a port 80 application port, and place it on the internal load balancer. In a two-node-type cluster, one node type is on the external load balancer. The other node type is on the internal load balancer. To use a two-node-type cluster, in the portal-created two-node-type template (which comes with two load balancers), switch the second load balancer to an internal load balancer. For more information, see the [Internal-only load balancer](#internallb) section. 1. Add the static internal load balancer IP address parameter. (For notes related to using a dynamic IP address, see earlier sections of this article.) ```json "internalLBAddress": { "type": "string", "defaultValue": "10.0.0.250" } ``` 2. Add an application port 80 parameter. 3. To add internal versions of the existing networking variables, copy and paste them, and add "-Int" to the name: ```json /* Add internal load balancer networking variables */ "lbID0-Int": "[resourceId('Microsoft.Network/loadBalancers', concat('LB','-', parameters('clusterName'),'-',parameters('vmNodeType0Name'), '-Internal'))]", "lbIPConfig0-Int": "[concat(variables('lbID0-Int'),'/frontendIPConfigurations/LoadBalancerIPConfig')]", "lbPoolID0-Int": "[concat(variables('lbID0-Int'),'/backendAddressPools/LoadBalancerBEAddressPool')]", "lbProbeID0-Int": "[concat(variables('lbID0-Int'),'/probes/FabricGatewayProbe')]", "lbHttpProbeID0-Int": "[concat(variables('lbID0-Int'),'/probes/FabricHttpGatewayProbe')]", "lbNatPoolID0-Int": "[concat(variables('lbID0-Int'),'/inboundNatPools/LoadBalancerBEAddressNatPool')]", /* Internal load balancer networking variables end */ ``` 4. If you start with the portal-generated template that uses application port 80, the default portal template adds AppPort1 (port 80) on the external load balancer. In this case, remove AppPort1 from the external load balancer `loadBalancingRules` and probes, so you can add it to the internal load balancer: ```json "loadBalancingRules": [ { "name": "LBHttpRule", "properties":{ "backendAddressPool": { "id": "[variables('lbPoolID0')]" }, "backendPort": "[parameters('nt0fabricHttpGatewayPort')]", "enableFloatingIP": "false", "frontendIPConfiguration": { "id": "[variables('lbIPConfig0')]" }, "frontendPort": "[parameters('nt0fabricHttpGatewayPort')]", "idleTimeoutInMinutes": "5", "probe": { "id": "[variables('lbHttpProbeID0')]" }, "protocol": "tcp" } } /* Remove AppPort1 from the external load balancer. { "name": "AppPortLBRule1", "properties": { "backendAddressPool": { "id": "[variables('lbPoolID0')]" }, "backendPort": "[parameters('loadBalancedAppPort1')]", "enableFloatingIP": "false", "frontendIPConfiguration": { "id": "[variables('lbIPConfig0')]" }, "frontendPort": "[parameters('loadBalancedAppPort1')]", "idleTimeoutInMinutes": "5", "probe": { "id": "[concate(variables('lbID0'), '/probes/AppPortProbe1')]" }, "protocol": "tcp" } }*/ ], "probes": [ { "name": "FabricGatewayProbe", "properties": { "intervalInSeconds": 5, "numberOfProbes": 2, "port": "[parameters('nt0fabricTcpGatewayPort')]", "protocol": "tcp" } }, { "name": "FabricHttpGatewayProbe", "properties": { "intervalInSeconds": 5, "numberOfProbes": 2, "port": "[parameters('nt0fabricHttpGatewayPort')]", "protocol": "tcp" } } /* Remove AppPort1 from the external load balancer. { "name": "AppPortProbe1", "properties": { "intervalInSeconds": 5, "numberOfProbes": 2, "port": "[parameters('loadBalancedAppPort1')]", "protocol": "tcp" } } */ ], "inboundNatPools": [ ``` 5. Add a second `Microsoft.Network/loadBalancers` resource. It looks similar to the internal load balancer created in the [Internal-only load balancer](#internallb) section, but it uses the "-Int" load balancer variables, and implements only the application port 80. This also removes `inboundNatPools`, to keep RDP endpoints on the public load balancer. If you want RDP on the internal load balancer, move `inboundNatPools` from the external load balancer to this internal load balancer: ```json /* Add a second load balancer, configured with a static privateIPAddress and the "-Int" load balancer variables. */ { "apiVersion": "[variables('lbApiVersion')]", "type": "Microsoft.Network/loadBalancers", /* Add "-Internal" to the name. */ "name": "[concat('LB','-', parameters('clusterName'),'-',parameters('vmNodeType0Name'), '-Internal')]", "location": "[parameters('computeLocation')]", "dependsOn": [ /* Remove public IP dependsOn, add vnet dependsOn "[concat('Microsoft.Network/publicIPAddresses/',concat(parameters('lbIPName'),'-','0'))]" */ "[concat('Microsoft.Network/virtualNetworks/',parameters('virtualNetworkName'))]" ], "properties": { "frontendIPConfigurations": [ { "name": "LoadBalancerIPConfig", "properties": { /* Switch from Public to Private IP address */ "publicIPAddress": { "id": "[resourceId('Microsoft.Network/publicIPAddresses',concat(parameters('lbIPName'),'-','0'))]" } */ "subnet" :{ "id": "[variables('subnet0Ref')]" }, "privateIPAddress": "[parameters('internalLBAddress')]", "privateIPAllocationMethod": "Static" } } ], "backendAddressPools": [ { "name": "LoadBalancerBEAddressPool", "properties": {} } ], "loadBalancingRules": [ /* Add the AppPort rule. Be sure to reference the "-Int" versions of backendAddressPool, frontendIPConfiguration, and the probe variables. */ { "name": "AppPortLBRule1", "properties": { "backendAddressPool": { "id": "[variables('lbPoolID0-Int')]" }, "backendPort": "[parameters('loadBalancedAppPort1')]", "enableFloatingIP": "false", "frontendIPConfiguration": { "id": "[variables('lbIPConfig0-Int')]" }, "frontendPort": "[parameters('loadBalancedAppPort1')]", "idleTimeoutInMinutes": "5", "probe": { "id": "[concat(variables('lbID0-Int'),'/probes/AppPortProbe1')]" }, "protocol": "tcp" } } ], "probes": [ /* Add the probe for the app port. */ { "name": "AppPortProbe1", "properties": { "intervalInSeconds": 5, "numberOfProbes": 2, "port": "[parameters('loadBalancedAppPort1')]", "protocol": "tcp" } } ], "inboundNatPools": [ ] }, "tags": { "resourceType": "Service Fabric", "clusterName": "[parameters('clusterName')]" } }, ``` 6. In `networkProfile` for the `Microsoft.Compute/virtualMachineScaleSets` resource, add the internal back-end address pool: ```json "loadBalancerBackendAddressPools": [ { "id": "[variables('lbPoolID0')]" }, { /* Add internal BE pool */ "id": "[variables('lbPoolID0-Int')]" } ], ``` 7. Deploy the template: ```powershell New-AzResourceGroup -Name sfnetworkinginternalexternallb -Location westus New-AzResourceGroupDeployment -Name deployment -ResourceGroupName sfnetworkinginternalexternallb -TemplateFile C:\SFSamples\Final\template\_internalexternalLB.json ``` After deployment, you can see two load balancers in the resource group. If you browse the load balancers, you can see the public IP address and management endpoints (ports 19000 and 19080) assigned to the public IP address. You also can see the static internal IP address and application endpoint (port 80) assigned to the internal load balancer. Both load balancers use the same virtual machine scale set back-end pool. ## Notes for production workloads The above GitHub templates are designed to work with the default SKU for Azure Standard Load Balancer (SLB), the Standard SKU. Any Service Fabric cluster using the Standard SKU for SLB needs to ensure that each node type has a rule allowing outbound traffic on port 443. This is necessary to complete cluster setup, and any deployment without such a rule will fail. In the above example of an "internal only" load balancer, an additional external load balancer must be added to the template with a rule allowing outbound traffic for port 443. ## Next steps [Create a cluster](service-fabric-cluster-creation-via-arm.md)