Proposed Pull Request Change

📄 Document Links

View on GitHub View on Microsoft Learn

--- title: "Tips and tools for troubleshooting Azure IoT Operations" description: Use common Kubernetes and MQTT tools such as kubectl, k9s, MQTT explorer, and mosquitto to troubleshoot and test your Azure IoT Operations instance. author: dominicbetts ms.author: dobett ms.service: azure-iot-operations ms.topic: how-to ms.date: 07/06/2026 --- # Tips and tools for troubleshooting your Azure IoT Operations instance This article describes how to use some common tools when you're learning, exploring, or troubleshooting your Azure IoT Operations instances. These tools are in addition to the capabilities provided by the Azure portal, Azure CLI, operations experience web UI, and [observability resources](../deploy-iot-ops/howto-configure-observability.md). ## Kubernetes tools Azure IoT Operations components run in a standard Kubernetes cluster. You can use the `kubectl` and `k9s` CLI tools to interact with and manage your cluster. ### Manage components using Kubernetes deployment manifests > [!IMPORTANT] > The use of Kubernetes deployment manifests is not supported in production environments and should only be used for debugging and testing. In general, Azure IoT Operations uses the Azure Arc platform to provide a hybrid cloud experience where you can manage the configuration through Azure Resource Manager (ARM) and front-end tools like the Azure portal, Bicep, and the Azure CLI. However, in a debug or test environment you can manage the components of Azure IoT Operations using YAML Kubernetes deployment manifests. This means you can use tools like `kubectl` to manage some components of Azure IoT Operations. This feature has some limitations: - Unless you enable resource sync in Azure IoT Operations using `az iot ops enable-rsync` command, changes made to the resources using Kubernetes deployment manifests are not synced to Azure. To learn more about resource sync, see [Resource sync](/azure/azure-arc/data/resource-sync). - Even if resource sync is enabled, brand new resources created using Kubernetes deployment manifests are not synced to Azure. Only changes to existing resources are synced. > [!IMPORTANT] > In production, the cloud is always the source of truth. Always create and modify resources through Azure—by using the operations experience, the Azure portal, the Azure CLI, or ARM/Bicep templates. Creating resources directly on the cluster or editing existing Kubernetes custom resources can cause the cloud and edge to go out of sync and isn't supported in production environments. ### `kubectl` `kubectl` is the Kubernetes command-line tool for managing your cluster. It has many capabilities that you can learn about in the official [kubernetes documentation](https://kubernetes.io/docs/reference/kubectl/introduction/). This article describes the common uses for `kubectl` when you're working with Azure IoT Operations such as listing the running pods and viewing logs. #### Configure `kubectl` to connect to your instance The [Prepare your Azure Arc-enabled Kubernetes cluster](../deploy-iot-ops/howto-prepare-cluster.md) article describes how to configure `kubectl` to connect to your **k3s** cluster when you run `kubectl` commands on the same machine where you deployed your Kubernetes cluster. > [!TIP] > Add the `export KUBECONFIG=~/.kube/config` command to your *.bashrc* or *.bash_profile* file so that you don't have to set the `KUBECONFIG` environment variable each time you open a new terminal window. If you deployed your Azure IoT Operations instance to an Arc-enabled AKS-EE, the `kubectl` configuration is automatically set up for you. You can run `kubectl` commands directly from the command line on the machine where you deployed your cluster. It's also possible to run `kubectl` commands from your local client machine rather than the machine where you deployed your Arc-enabled cluster: ### [Ubuntu/k3s](#tab/ubuntu) As a one-time step, use SSH to connect to the machine where you deployed your cluster and run the following commands. Be sure to replace `<your-name>` with your name: ```bash kubectl create serviceaccount <your-name> -n default kubectl create clusterrolebinding <your-name>-binding --clusterrole cluster-admin --serviceaccount default:<your-name> kubectl apply -f - <<EOF apiVersion: v1 kind: Secret metadata: name: <your-name>-secret annotations: kubernetes.io/service-account.name: <your-name> type: kubernetes.io/service-account-token EOF TOKEN=$(kubectl get secret <your-name>-secret -o jsonpath='{$.data.token}' | base64 -d | sed 's/$/\n/g') echo $TOKEN ``` ### [AKS](#tab/aks) As a one-time step, open a PowerShell prompt on the machine where you deployed your cluster and run the following commands. Be sure to replace `<your-name>` with your name: ```powershell kubectl create serviceaccount "<your-name>" -n default kubectl create clusterrolebinding "<your-name>-binding" --clusterrole cluster-admin --serviceaccount default:"<your-name>" $yaml = @" apiVersion: v1 kind: Secret metadata: name: <your-name>-secret annotations: kubernetes.io/service-account.name: <your-name> type: kubernetes.io/service-account-token "@ $yaml | kubectl apply -f - $TOKEN = ([System.Text.Encoding]::UTF8.GetString([System.Convert]::FromBase64String((kubectl get secret <your-name>-secret -o jsonpath='{$.data.token}')))) Write-Output $TOKEN ``` --- Make a note of the token. You use this token to authenticate when you run `kubectl` commands on your client machine. You can now disconnect from the machine running your Kubernetes cluster. To use `kubectl` on your client machine to connect to the cluster, open two terminals: 1. In the first terminal, run the following command to configure a proxy to connect to your cluster. Be sure to replace the values for the three placeholders: ```console az connectedk8s proxy -n <your-arc-enabled-cluster-name> -g <your-arc-enabled-cluster-resource-group> --token <token-from-previous-step> ``` Leave this terminal open while you run `kubectl` commands in the second terminal. 1. In the second terminal, you can run your `kubectl` commands against your remote cluster. For example, to list the pods in the `azure-iot-operations` namespace: ```console kubectl get pods -n azure-iot-operations ``` > [!TIP] > You can also run commands, such as `k9s`, that use the `kubectl` configuration in this terminal. The `kubectl` context remains set to the remote cluster until you close your first terminal. To learn more, see [Use cluster connect to securely connect to Azure Arc-enabled Kubernetes clusters](/azure/azure-arc/kubernetes/cluster-connect). #### Namespaces By default, Arc and Azure IoT Operations use the following namespaces in the Kubernetes cluster: - `azure-iot-operations` for the Azure IoT Operations components. - `azure-arc` for the Azure Arc-enabled Kubernetes components. > [!TIP] > To view all the namespaces in your cluster, run the following command: `kubectl get namespaces`. #### Common `kubectl` commands To view all the pods running in the `azure-iot-operations` namespace, run the following command: ```bash kubectl get pods -n azure-iot-operations ``` The output looks like the following example: ```output NAME READY STATUS RESTARTS AGE adr-schema-registry-0 2/2 Running 0 19m adr-schema-registry-1 2/2 Running 0 19m aio-akri-agent-777477bc68-72lrg 1/1 Running 7 (83m ago) 21d aio-broker-authentication-0 1/1 Running 7 (83m ago) 21d aio-broker-backend-1-0 1/1 Running 11 (82m ago) 21d aio-broker-backend-1-1 1/1 Running 7 (83m ago) 21d aio-broker-diagnostics-probe-0 1/1 Running 11 (83m ago) 21d aio-broker-diagnostics-service-0 1/1 Running 7 (83m ago) 21d aio-broker-fluent-bit-6bkf2 1/1 Running 0 16m aio-broker-frontend-0 1/1 Running 12 (83m ago) 21d aio-broker-health-manager-0 1/1 Running 14 (82m ago) 21d aio-broker-operator-0 1/1 Running 7 (83m ago) 21d aio-broker-upgrade-status-job-1.0.4-bwlcc 0/1 Completed 0 77m aio-broker-webhook-admission-65d67f8ddc-jct9j 1/1 Running 0 82m aio-dataflow-admission-webhook-84dd44c8bd-6pw58 1/1 Running 7 (83m ago) 21d aio-dataflow-operator-0 1/1 Running 14 (83m ago) 21d aio-dataflow-upgrade-status-job-1.0.5-msmf4 0/1 Completed 0 77m aio-opc-asset-discovery-54649d46cf-kb6qs 1/1 Running 2 (83m ago) 17d aio-opc-media-1-785748ff6c-qkhgl 1/1 Running 1 (83m ago) 14d aio-opc-opc.tcp-1-858b9ff67-dxwvb 1/1 Running 4 (80m ago) 17d aio-opc-supervisor-5d6b9bfc49-fgt7d 1/1 Running 2 (83m ago) 17d aio-operator-7b9b585dc6-bvfpd 2/2 Running 0 19m aio-usage-28946280-f42k8 0/1 Completed 0 14d aio-usage-28946340-45grx 0/1 Completed 0 14d aio-usage-28946400-znn7v 0/1 Completed 0 13d aio-usage-28946460-nrw4z 0/1 Completed 0 13d aio-usage-28966500-mrcmf 0/1 Completed 0 55m ``` To view the [logs](https://kubernetes.io/docs/reference/kubectl/generated/kubectl_logs/) for a specific pod such as the *aio-opc-opc.tcp-1-858b9ff67-dxwvb* pod, run the following command: ```bash kubectl logs aio-opc-opc.tcp-1-858b9ff67-dxwvb -n azure-iot-operations ``` To view a human-readable [description](https://kubernetes.io/docs/reference/kubectl/generated/kubectl_describe/) of a specific pod such as the *aio-opc-opc.tcp-1-858b9ff67-dxwvb* pod, run the following command: ```bash kubectl describe pod aio-opc-opc.tcp-1-858b9ff67-dxwvb -n azure-iot-operations ``` In some places, the Azure IoT Operations documentation uses the [kubectl apply](https://kubernetes.io/docs/reference/kubectl/generated/kubectl_apply/) command to apply a Kubernetes manifest file to make a configuration change in the cluster. ### `k9s` The [k9s](https://k9scli.io/) utility offers a terminal-based UI for managing your Kubernetes cluster. It uses your `kubectl` configuration to connect to your cluster and provides a visual way to interact with your cluster. Its default view lists all the pods currently running in your cluster: :::image type="content" source="media/tips-tools/default-k9s-view.png" alt-text="Screenshot that shows the default k9s view."::: When you're working with Azure IoT Operations, you can filter the view to show only the pods in the `azure-iot-operations` namespace. 1. Type `:` to open the command panel, then type `ns` and press **Enter**. 1. In the list of namespaces, select `azure-iot-operations` and press **Enter**. 1. The list of pods now shows only the pods in the **azure-iot-operations** namespace: :::image type="content" source="media/tips-tools/filtered-k9s-view.png" alt-text="Screenshot that shows the k9s list of pods filtered to the azure-iot-operations namespace."::: > [!TIP] > You can now use the number keys to apply filters. The previous screenshot shows that **0** shows all pods, and **1** shows only the pods in the **azure-iot-operations** namespace. You can use shortcut keys to view information about your pods. For example: - To describe a pod, select it in the list and press **d**. :::image type="content" source="media/tips-tools/describe-k9s-view.png" alt-text="Screenshot that shows a description in k9s of a running pod."::: - To view the logs for a pod, select it in the list and press **l**. :::image type="content" source="media/tips-tools/logs-k9s-view.png" alt-text="Screenshot that shows the log of a running pod in k9s."::: > [!TIP] > You can use the number keys to navigate around the log file. To view custom resource types other that pods in the cluster: 1. Press **Ctrl-a** to display the list of custom resource types. 1. Select the custom resource type, such as **devices** and press **Enter**. > [!TIP] > To search for a custom resource type by name, type **/** and then start typing the name of the type you're looking for. 1. Select a custom resource and choose one of the available operations. For example, you can view the YAML definition of a device endpoint profile by selecting it and pressing **y**. For some resources, you can edit the configuration. The following table describes some of the custom resource types you might work with in Azure IoT Operations: | Custom resource type | Description | |----------------------|-------------| | `devices` | Represents the configuration for a device. | | `assets` | Represents the configuration for an asset. | | `brokers`, `brokerlisters`, `brokerauthentications`, `brokerauthorizations` | Represent the configuration for an MQTT broker. | | `dataflows`, `dataflowendpoints`, `dataflowprofiles` | Represent the configuration for a data flow. | | `secrets`, `secretsyncs`, `secretproviderclasses` | Represent the configuration for secrets and secret management. | ## MQTT tools To interact with the MQTT broker in your Azure IoT Operations instance for learning, testing, or troubleshooting, first expose the broker by using one of the nonproduction approaches described in [Test connectivity to MQTT broker with MQTT clients](../manage-mqtt-broker/howto-test-connection.md): connect from inside the cluster, or expose the broker outside the cluster with a `NodePort` or `LoadBalancer` service. The rest of this section shows the troubleshooting tools you can run against those test listeners. > [!CAUTION] > These approaches are only suitable for development and test environments. Never use them in production. ### Connect to the default listener inside the cluster Deploy the `mqtt-client` pod and open a shell in it as described in [Connect to the default listener inside the cluster](../manage-mqtt-broker/howto-test-connection.md#connect-to-the-default-listener-inside-the-cluster). Then, from the shell, use `mosquitto_sub` and `mosquitto_pub` to inspect and publish messages on the Azure IoT Operations data topics. For example, to subscribe to all topics under `azure-iot-operations/data`: ```console mosquitto_sub --host aio-broker --port 18883 --topic "azure-iot-operations/data/#" --verbose --cafile /var/run/certs/ca.crt -D CONNECT authentication-method 'K8S-SAT' -D CONNECT authentication-data $(cat /var/run/secrets/tokens/broker-sat) ``` To receive a single message from the `azure-iot-operations/data/thermostat` topic, add the `-C 1` option: ```console mosquitto_sub --host aio-broker --port 18883 --topic "azure-iot-operations/data/thermostat" -C 1 --verbose --cafile /var/run/certs/ca.crt -D CONNECT authentication-method 'K8S-SAT' -D CONNECT authentication-data $(cat /var/run/secrets/tokens/broker-sat) ``` To view the MQTT v5 user properties in the messages, use the `-F %P` option: ```console mosquitto_sub --host aio-broker --port 18883 --topic "azure-iot-operations/data/thermostat" -V mqttv5 -F %P --cafile /var/run/certs/ca.crt -D CONNECT authentication-method 'K8S-SAT' -D CONNECT authentication-data $(cat /var/run/secrets/tokens/broker-sat) ``` To publish a message to the `azure-iot-operations/data/valve` topic: ```console mosquitto_pub --host aio-broker --port 18883 --topic "azure-iot-operations/data/valve" --message "open:15%" --id "controller" --cafile /var/run/certs/ca.crt -D CONNECT authentication-method 'K8S-SAT' -D CONNECT authentication-data $(cat /var/run/secrets/tokens/broker-sat) ``` When you finish, delete the client pod: ```console kubectl delete pod mqtt-client -n azure-iot-operations ``` ### Use a `NodePort` or `LoadBalancer` service If you follow the steps to configure a [Node port](../manage-mqtt-broker/howto-test-connection.md#node-port) or [Load balancer](../manage-mqtt-broker/howto-test-connection.md#load-balancer) service to expose the MQTT broker outside of the cluster, you can use any MQTT client tools that support connecting to a specific port. The following examples assume that you configured the service with no authentication, authorization, or TLS. You can now use your favorite MQTT client tools to connect to the MQTT broker on port 1883 if you're using a load balancer or the configured port if you're using a node port. For example, to run the open-source [mqttui](https://github.com/EdJoPaTo/mqttui) tool on the machine where your Kubernetes cluster is running, use the following command: ```console mqttui --broker mqtt://localhost:1883 ``` > [!TIP] > If you configured a load balancer and port 1883 is open on the host machine's public IP address, you can use the following command to connect to the MQTT broker from a different machine: `mqttui --broker mqtt://<cluster-machine-public-ip>:1883` You can use the `mqttui` tool to subscribe to topics, publish messages, and view the messages that are flowing through the broker: :::image type="content" source="media/tips-tools/mqttui-all-topics.png" alt-text="Screenshot that shows the MQTTUI tool displaying all topics."::: To display the messages on a specific topic such as `azure-iot-operations/data/thermostat`, use the following command: ```console mqttui --broker mqtt://localhost:1883 azure-iot-operations/data/thermostat ``` To publish a message to the `azure-iot-operations/data/valve` topic, use the following command: ```console mqttui publish --broker mqtt://localhost:1883 azure-iot-operations/data/valve open:15% ``` To run the open-source [MQTT Explorer](https://mqtt-explorer.com/) tool on the machine where your Kubernetes cluster is running, use the following configuration: :::image type="content" source="media/tips-tools/mqtt-explorer-localhost.png" alt-text="Screenshot that shows the MQTT Explorer localhost configuration."::: To run the open-source [MQTT Explorer](https://mqtt-explorer.com/) tool on your local machine to connect to the machine where your Kubernetes cluster is running, use the following configuration: :::image type="content" source="media/tips-tools/mqtt-explorer-remote-host.png" alt-text="Screenshot that shows the MQTT Explorer remote host configuration."::: Make sure that MQTT Explorer has at least the `#` topic configured: :::image type="content" source="media/tips-tools/mqtt-explorer-topic-configuration.png" alt-text="Screenshot that shows the MQTT Explorer default topic configuration."::: After you connect, you can see messages in the topics you subscribed to and publish messages: :::image type="content" source="media/tips-tools/mqtt-explorer-subscription.png" alt-text="Screenshot that shows the MQTT Explorer subscribed to Azure IoT Operations topics."::: ## Tips Here are some additional tips to help you work with your Azure IoT Operations instance: ### Find the custom location of your Azure IoT Operations instance To find the custom location associated with your Azure IoT Operations instance, use the following command: ```azurecli az iot ops show --name <YOUR_INSTANCE_NAME> --resource-group <YOUR_RESOURCE_GROUP> --query "extendedLocation.name" --output tsv ``` You can also find the custom location in the Azure portal on the instance overview page in the **Extended location** field.