Proposed Pull Request Change

📄 Document Links

View on GitHub View on Microsoft Learn

--- title: Schedule and Deploy Batch Jobs with Kueue on Azure Kubernetes Service (AKS) description: Learn how to define Kueue deployments and efficiently schedule batch workloads on your Azure Kubernetes Service (AKS) cluster. ms.topic: how-to ms.service: azure-kubernetes-service ms.date: 9/26/2025 author: colinmixonn ms.author: colinmixon # Customer intent: "As a platform admin, I want to quickly schedule and deploy batch jobs to ensure efficient resource utilization and cost optimization on AKS cluster(s), enabling platform developers to run bursty workloads to completion without impacting the performance of other services." --- # Schedule and deploy batch jobs with Kueue on Azure Kubernetes Service (AKS) In this article, you learn how to schedule and deploy sample batch jobs on Azure Kubernetes Service (AKS) using Kueue. Also, this guide covers installing Kueue, configuring ResourceFlavors and ClusterQueues for fine-grained resource management, and submitting jobs via LocalQueues. You also learn how to use Kueue to queue up a sample batch job and track the results across Pending, Running, and Finished states. [!INCLUDE [open source disclaimer](./includes/open-source-disclaimer.md)] To learn more about Kueue and common uses cases for batch workload administrators and users, see [Kueue overview on AKS](./kueue-overview.md). ## Prerequisites * An existing AKS cluster. If you don't have a cluster, create one using the [Azure CLI][aks-quickstart-cli], [Azure PowerShell][aks-quickstart-powershell], or the [Azure portal][aks-quickstart-portal]. * Azure CLI installed on your local machine. To install or upgrade, see [Install the Azure CLI](/cli/azure/install-azure-cli). * [Helm version 3 or above](https://helm.sh/docs/intro/install/) installed. * [The latest version of Kueue installed in a dedicated namespace on your cluster](./kueue-overview.md#prerequisites). ## Define a ResourceFlavor object In Kueue, a ResourceFlavors enables fine-grained resource management by associating workloads with specific nodes, taints, tolerations, or availability zones. For nodes, `ResourceFlavors` can define the characteristics like pricing, availability, brands, models, and architecture (that is, x86 versus ARM CPUs). A `ClusterQueue` uses these flavors to manage quotas and admission policies for workloads. This configuration defines a `ResourceFlavor` without any labels or taints, known as an empty `ResourceFlavor`. This configuration is perfect when quotas for different flavors don't need to be managed. 1. Create and save a `ResourceFlavor` in a file named `resourceflavor-sample.yaml` with the following manifest: ```bash cat << EOF > resourceflavor-sample.yaml apiVersion: kueue.x-k8s.io/v1beta1 kind: ResourceFlavor metadata: name: on-demand EOF ``` 2. apply ```bash kubectl apply -f resourceflavor-sample.yaml ``` 3. verify ```bash kubectl get resourceflavors ``` Example output ```output NAME AGE on-demand 5m32s ``` ## Create a ClusterQueue A ClusterQueue is a cluster-scoped resource that governs a pool of resources, defining usage limits and Fair Sharing rules. Where applicable, Fair Sharing rules allow another ClusterQueue in the **same** cohort to unused quota for pending jobs. Each ClusterQueue specifies which flavors it supports and how much quota is available for each. This sample `ClusterQueue` defines: * **`namespaceSelector: {}`**: Indicates that `sample-jobs` accepts workloads from any namespace that references this `ClusterQueue` via a `LocalQueue` (you can restrict usage (for example, to only team A's namespace) with a label selector). * **`coveredResources: ["cpu", "memory"]` in `resourceGroups`**: Defines the standard CPU and memory resource types managed by this `ClusterQueue`. * **`flavor` of `on-demand` nodes with `4` CPUs, `8Gi` memory**: Only workloads scheduled on `on-demand` nodes consume this quota. If the cluster uses up this quota, it doesn't admit any other workloads using this flavor (unless you allow borrowing from the `cohort`). 1. Create and save a Kueue `ClusterQueue` in a file named `clusterqueue-sample.yaml` with the following manifest: ```batch cat <<EOF > clusterqueue-sample.yaml apiVersion: kueue.x-k8s.io/v1beta1 kind: ClusterQueue metadata: name: sample-jobs spec: cohort: general namespaceSelector: {} # Accept workloads from any namespace resourceGroups: - coveredResources: ["cpu", "memory"] flavors: - name: on-demand resources: - name: "cpu" nominalQuota: 4 - name: "memory" nominalQuota: 8Gi EOF ``` 2. Apply the `ClusterQueue` manifest using the `kubectl apply` command. ```bash kubectl apply -f clusterqueue-sample.yaml ``` 3. Verify the ClusterQueue` manifest was applied ```bash kubectl get clusterqueues ``` Example output ```output NAME COHORT PENDING WORKLOADS sample-jobs general 0 ``` > [!NOTE] > The `ClusterQueue` isn't ready for use until a `ResourceFlavor` object is configured. If you create a `ClusterQueue` without any existing `ResourceFlavor`, workloads referencing it are marked as `Inadmissible`. ## Create a LocalQueue A LocalQueue is a namespace-scoped resource that acts as a gateway for users to submit jobs. A `LocalQueue` is assigned to one `ClusterQueue` from which resources are allocated to run its workloads. This sample `LocalQueue` configures the following settings: * Enables users in the `batch-jobs` namespace to submit batch workloads to Kueue. * Route the batch workloads to the `sample-jobs` `ClusterQueue`, which manages the actual compute resource quotas and scheduling policies. 1. Create a namespace named *batch-jobs* using the `kubectl create` command. ```bash kubectl create ns batch-jobs ``` 2. Create and save a `LocalQueue` in a file named `localqueue-sample.yaml` with the following YAML manifest: ```bash cat <<EOF > localqueue-sample.yaml apiVersion: kueue.x-k8s.io/v1beta1 kind: LocalQueue metadata: name: sample-queue namespace: batch-jobs spec: clusterQueue: sample-jobs EOF ``` 3. Apply the `LocalQueue` manifest using the `kubectl apply` command. ```bash kubectl apply -f localqueue-sample.yaml ``` 4. Verify the `LocalQueue` manifest was applied ```bash kubectl get localqueues --all-namespaces ``` Exampmle output ```output NAMESPACE NAME CLUSTERQUEUE PENDING WORKLOADS ADMITTED WORKLOADS batch-jobs sample-queue sample-jobs 0 0 ``` ## Create 2 batch jobs This configuration defines two Kubernetes batch jobs submitted to the batch-jobs namespace and assigned to the sample-queue managed by Kueue. Both jobs are single-instance (parallelism: 1, completions: 1) and are configured with `Never` restart policy. The fields `parallelism` and `completions` control how many pods are run and how the job is considered complete. So `parallelism` and `completions` of 1 means that one pod can run at once, and the job is marked as complete once one pod finishes successfully, per batch job. * Job test-batch-1: Requests one CPU and 500Mi memory * Job test-batch-2: Requests two CPUs and 1Gi memory 1. Create two sample batch jobs to deploy in the *batch-jobs* namespace using the following YAML manifest named `batch-workloads.yaml`: ```bash cat <<EOF > batch-workloads.yaml apiVersion: batch/v1 kind: Job metadata: name: test-batch-1 namespace: batch-jobs labels: kueue.x-k8s.io/queue-name: sample-queue spec: parallelism: 1 completions: 1 template: spec: containers: - name: dummy-job image: registry.k8s.io/e2e-test-images/agnhost:2.53 command: ["sh", "-c", "echo Running test-batch-1; sleep 60"] resources: requests: cpu: "1" memory: "500Mi" limits: cpu: "1" memory: "500Mi" restartPolicy: Never --- apiVersion: batch/v1 kind: Job metadata: name: test-batch-2 namespace: batch-jobs labels: kueue.x-k8s.io/queue-name: sample-queue spec: parallelism: 1 completions: 1 template: spec: containers: - name: dummy-job image: registry.k8s.io/e2e-test-images/agnhost:2.53 command: ["sh", "-c", "echo Waiting in queue for CPUs...; sleep 30"] resources: requests: cpu: "2" memory: "1Gi" limits: cpu: "2" memory: "1Gi" restartPolicy: Never EOF ``` 2. Apply the manifest for the batch jobs using the `kubectl apply` command. ```bash kubectl apply -f batch-workloads.yaml ``` ## Verify Batch Jobs are Submitted to `LocalQueue` 1. View the status of the batched workloads using the `kubectl get` command. ```bash kubectl get workloads --namespace batch-jobs ``` Example output ```output NAME ADMITTED AGE test-batch-1 True 10s test-batch-2 False 5s ``` 2. Run the following command for `test-batch-2` while it is in a `Pending` state ```bash kubectl get workloads test-batch-2 -o yaml ``` Expected output ```output ... ... Status: Conditions: Type: Admitted Status: False Reason: QuotaUnavailable Message: Insufficient quota in ClusterQueue sample-jobs (flavor on-demand): requested 2 CPUs, available 1 ... ... ``` After `test-batch-1` completes, `test-batch-2` will be admitted and run. Now, the output should look like the following example output: ```output Status: Conditions: Type: Admitted Status: True Last Transition Time: 1234-56-78T00:00:00Z Admission: ClusterQueue: sample-jobs PodSetAssignments: Name: main Flavors: cpu: on-demand memory: on-demand ResourceUsage: cpu: 2 memory: 1Gi ``` 3. View the final status of the `batch-jobs` namespace using the `kubectl get` command. ```bash kubectl get job,deploy,rs,pod,workload --namespace batch-jobs ``` Example output ```output NAME STATUS COMPLETIONS DURATION AGE job.batch/test-batch-1 Complete 1/1 97s 3m15s job.batch/test-batch-2 Complete 1/1 35s 3m15s NAME READY STATUS RESTARTS AGE pod/test-batch-1-hb8zl 0/1 Completed 0 3m15s pod/test-batch-2-dx9hk 0/1 Completed 0 3m15s NAME QUEUE RESERVED IN ADMITTED FINISHED AGE workload.kueue.x-k8s.io/job-test-batch-1-6fb85 sample-queue sample-jobs True True 3m15s workload.kueue.x-k8s.io/job-test-batch-2-84f49 sample-queue sample-jobs True True 3m15s ``` ## FAQ ### Question 1: How can I confirm that the Kueue controller is available and running as expected? 1. Confirm the Kueue controller manager pod is running using the `kubectl get` command. ```bash kubectl get pods --namespace kueue-system ``` The Kueue controller manager pod should be in a `Running` state with `1/1` containers ready, as shown in the following example output: ```output NAME READY STATUS RESTARTS AGE kueue-controller-manager-xxxxxxx 1/1 Running 0 2m ``` 2. If the `Status` shows `CrashLoopBackOff` or `Pending`, check the deployment logs using the `kubectl logs` command. ```bash kubectl logs --namespace kueue-system deployment/kueue-controller-manager ``` ### Question 2: One or more of the Kueue custom resources (CRDs) are missing when I install via Helm. How can I ensure all of the CRDs are installed? 1. After installing Kueue with the [Kueue overview on AKS](./kueue-overview.md) guidance, confirm that all of the CRDs are installed using the `kubectl get` command. ```bash kubectl get crds | grep kueue ``` These CRDs should be listed, as shown in the following example output: ```output admissionchecks.kueue.x-k8s.io clusterqueues.kueue.x-k8s.io cohorts.kueue.x-k8s.io localqueues.kueue.x-k8s.io multikueueclusters.kueue.x-k8s.io multikueueconfigs.kueue.x-k8s.io provisioningrequestconfigs.kueue.x-k8s.io resourceflavors.kueue.x-k8s.io topologies.kueue.x-k8s.io workloadpriorityclasses.kueue.x-k8s.io workloads.kueue.x-k8s.io ``` 2. If one or more of the CRDs are missing, you might see errors in controller logs, failed job queuing, `CrashLoopBackOff` for the controller, or inability to admit or schedule workloads. In this case, you can manually reinstall the Kueue CRDs using the `kubectl apply` command. ```bash kubectl apply -f https://github.com/kubernetes-sigs/kueue/releases/latest/download/kueue-crds.yaml ``` > [!NOTE] > Note that if you manually install the CRDs, you need to manually delete them once you're finished using the `kubectl delete` command. ### Question 3: What's the difference between a LocalQueue and a ClusterQueue A ClusterQueue is a cluster-scoped resource that defines and governs a pool of compute resources like CPU, memory, pods, and accelerators across the entire Kubernetes cluster. A LocalQueue is a namespace-scoped resource that acts as a gateway for users to submit jobs within the defined Kubernetes cluster. This separation allows for fine-grained control over resource allocation and multi-tenant scheduling without exposing cluster-wide quotas directly to users. How they work together: 1. A user submits a job to a LocalQueue in their namespace. 2. Kueue routes the job to the referenced ClusterQueue. 3. The ClusterQueue checks resource availability and quota limits. 4. If admitted, the job is unsuspended and scheduled. ## Next steps In this article, you: * Installed Kueue on your Azure Kubernetes Service (AKS) cluster using Helm and verified CRDs, controller health, and namespace setup. * Configured `ClusterQueue` and `LocalQueue` for general-purpose workloads with resource quotas and flavors (such as on-demand). * Submitted two batch jobs to demonstrate queuing: one admitted immediately, the second held due to quota limits, then admitted when resources became available. * Monitored workload status and controller logs to confirm scheduling behavior and queuing logic. To learn more about Kueue, visit the following resources: * [Multi-cluster scheduling and resource placement with Kueue and KubeFleet on AKS](https://blog.aks.azure.com/2025/04/02/Scaling-Kubernetes-for-AI-and-Data-intensive-Workloads). * [Kueue developer tools](https://kueue.sigs.k8s.io/docs/tasks/dev/) official documentation. <!-- LINKS --> [az-aks-get-credentials]: /cli/azure/aks#az-aks-get-credentials [aks-quickstart-cli]: ./learn/quick-kubernetes-deploy-cli.md [aks-quickstart-portal]: ./learn/quick-kubernetes-deploy-portal.md [aks-quickstart-powershell]: ./learn/quick-kubernetes-deploy-powershell.md