Azure Pipelines

Scale applications based on agent pool queues for Azure Pipelines.

Availability: v2.3+ Maintainer: Microsoft

Warning

You are currently viewing v2.21 of the documentation and it is not the latest. For the most recent documentation, kindly click here.

Trigger Specification

This specification describes the azure-pipelines trigger for Azure Pipelines. It scales based on the amount of pipeline runs pending in a given agent pool.

triggers:
  - type: azure-pipelines
    metadata:
      # Optional: Name of the pool in Azure DevOps
      poolName: "{agentPoolName}"
      # Optional: Learn more in 'How to determine your pool ID'
      poolID: "{agentPoolId}"
      # Optional: Azure DevOps organization URL, can use TriggerAuthentication as well
      organizationURLFromEnv: "AZP_URL"
      # Optional: Azure DevOps Personal Access Token, can use TriggerAuthentication as well
      personalAccessTokenFromEnv: "AZP_TOKEN"
      # Optional: Target queue length
      targetPipelinesQueueLength: "1" # Default 1
      activationTargetPipelinesQueueLength: "5" # Default 0
      # Optional: Parent template to read demands from
      parent: "{parent ADO agent name}"
      # Optional: Demands string to read demands from ScaledObject
      demands: "{demands}"
      # Optional: Demands of jobs must exactly match the capabilities the trigger defines
      requireAllDemands: false
      # Optional: Require jobs to include specified demands, ignoring any extra ones
      requireAllDemandsAndIgnoreOthers: false
      # Optional: How many jobs to fetch for the pool in the API (default: 250)
      jobsToFetch: "{jobsToFetch}"
      # Optional: Whether to only fetch unfinished jobs from the API (default: false)
      fetchUnfinishedJobsOnly: false
      # Optional: Property to enable case-insensitive comparison of pipeline job demands (default: false)
      caseInsensitiveDemandsProcessing: false
    authenticationRef:
     name: pipeline-trigger-auth

Parameter list:

poolName - Name of the pool. (Optional, either poolID or poolName must be configured)
poolID - Id of the pool. (Optional, either poolID or poolName must be configured)
organizationURLFromEnv - Name of the environment variable your deployment uses to get the URL for your Azure DevOps organization.
personalAccessTokenFromEnv - Name of the environment variable that provides the personal access token (PAT) for Azure DevOps. Learn more about how to create one in the official docs.
targetPipelinesQueueLength - Target value for the amount of pending jobs in the queue to scale on. (Default: 1, Optional)
- Example - If one pod can handle 10 jobs, set the queue length target to 10. If the actual number of jobs in the queue is 30, the scaler scales to 3 pods.
activationTargetPipelinesQueueLength - Target value for activating the scaler. Learn more about activation here. (Default: 0, Optional)
parent - Put the name of the ADO agent that matched the ScaledObject. e.g. mavenagent-scaledobject may have an initial deployment called “mavenagent-keda-template”; this is the deployment that is made offline. This name is provided to the initial deployment as the environment variable “AZP_NAME”. Mutually exclusive with jobsToFetch.
demands - Put the demands string that was provided to the ScaledObject. This MUST be a subset of the actual capability list the agent has. e.g. maven,docker
jobsToFetch - The number of the jobs that KEDA will fetch for the pool from Azure Pipeline API. Mutually exclusive with parent and fetchUnfinishedJobsOnly. (Default: 250, Optional)
fetchUnfinishedJobsOnly - Whether to fetch only unfinished jobs from the Azure Pipeline API. Normally both finished, running and pending jobs are returned by the API. When this parameter is set to true, the API call is modified so that only running and pending jobs are returned from the API, which reduces the amount of returned jobs considerably. Mutually exclusive with jobsToFetch. (Default: false, Optional)
caseInsensitiveDemandsProcessing - Property to enable case-insensitive comparison of pipeline job demands. When this parameter is set to true, the demands check is case-insensitive. (Default: false, Optional)

💡 NOTE: You can either use poolID or poolName. If both are specified, then poolName will be used.

Authentication Parameters

The Azure Pipelines scaler supports the following Azure DevOps authentication methods:

Personal Access Token (PAT)
Azure AD Workload Identity
Microsoft Entra service principal using a client secret or certificate

The Azure DevOps organization URL can be supplied directly through organizationURL, from the scale target through organizationURLFromEnv, or as an authentication parameter. A PAT can be supplied through personalAccessTokenFromEnv or an authenticationRef. Workload identity and service principal authentication must be configured through an authenticationRef.

When more than one authentication method is configured, KEDA uses the following precedence:

Personal Access Token
Azure AD Workload Identity
Microsoft Entra service principal

Personal Access Token authentication

organizationURL - URL of the Azure DevOps organization.
personalAccessToken - Personal Access Token for Azure DevOps.

Azure AD Workload Identity authentication

Configure a TriggerAuthentication or ClusterTriggerAuthentication with podIdentity.provider set to azure-workload.

Microsoft Entra service principal authentication

Configure the reusable azureServicePrincipal authentication provider with either a client secret or client certificate. The Azure Pipelines scaler acquires a Microsoft Entra token for Azure DevOps and sends it as a bearer token.

Before using the scaler, add the service principal to the Azure DevOps organization and grant it permission to read the required agent pool and its job requests. When adding it to Azure DevOps, use the service principal object ID from Enterprise applications, not the application registration object ID. For detailed setup instructions, see the Azure DevOps service principal documentation.

How to determine your pool ID

There are several ways to get the poolID. The easiest could be using az cli to get it using the command az pipelines pool list --pool-name {agentPoolName} --organization {organizationURL} --query [0].id.

It is also possible to get the pool ID using the UI by browsing to the agent pool from the organization (Organization settings -> Agent pools -> {agentPoolName}) and getting it from the URL. The URL should be similar to https://dev.azure.com/{organization}/_settings/agentpools?poolId={poolID}&view=jobs

Careful - You should determine this on an organization-level, not project-level. Otherwise, you might get an incorrect id.

Finally, it is also possible get the pool ID from the response of a HTTP request by calling the https://dev.azure.com/{organizationName}/_apis/distributedtask/pools?poolname={agentPoolName} endpoint in the key value[0].id.

Supporting demands in agents

By default, if you do not use demands in your agent scaler then it scales based on the pool’s queue length.

Demands (Capabilities) are useful when you have multiple agents with different capabilities existing within the same pool, for instance in a kube cluster you may have an agent supporting dotnet5, dotnet6, java or maven; particularly these would be exclusive agents where jobs would fail if run on the wrong agent. This is Microsoft’s demands feature.

Using Parent: Azure DevOps is able to determine which agents can match any job it is waiting for. If you specify a parent template then KEDA will further interrogate the job request to determine if the parent is able to fulfill the job. If the parent is able to complete the job it scales the workload fulfill the request. The parent template that is generally offline must stay in the Pool’s Agent list.
Using demands: KEDA will determine which agents can fulfill the job based on the demands provided. The demands are provided as a comma-separated list and must be a subset of the actual capabilities of the agent. (For example maven,java,make. Note: Agent.Version is ignored).
- If requireAllDemands is set to true, the job’s demands must match exactly with the triggers demands. This means a job with demands maven will not match an agent with capabilities maven,java.
- If requireAllDemandsAndIgnoreOthers is set to true, the job’s demands must include all specified demands exactly, but any additional demands will be ignored. For instance, a job with demands maven will match an agent with capabilities maven,java, as long as maven is in the demand list.
- Note: requireAllDemands takes precedence over requireAllDemandsAndIgnoreOthers. If both are set to true, only exact matches are accepted.

Microsoft’s documentation: https://learn.microsoft.com/en-us/azure/devops/pipelines/process/demands?view=azure-devops&tabs=yaml

Please note that the parent template feature is exclusive to KEDA and not Microsoft and is another way of supporting demands.

If you wish to use demands in your agent scaler then you can do so by adding the following to your pipeline:

    pool:
      - name: "{agentPoolName}"
        demands:
          - example-demands
          - another-demand -equals /bin/executable

Then, you can use the demands parameter to specify the demands that your agent supports or the parent parameter to link a template that matches you scaled object.

KEDA will use the following evaluation order:

If neither parent nor demands are defined in the scaling definition, it will scale the workload to fulfill the job.
If parent is set, KEDA will interrogate the job request to determine if the parent is able to fulfill the job. If the parent is able to complete the job it scales the workload to fulfill the request.
Finally, if the demands are set in the scaling definition then KEDA will determine which agents can fulfill the job based on the demands provided.

Note: If more than one scaling definition is able to fulfill the demands of the job then they will both spin up an agent.

How it works under the hood

Azure DevOps has a Job Request API with returns a list of all jobs, and the agent that they are assigned to, or could potentially be assigned to. This is an undocumented Microsoft API which is available on https://dev.azure.com/<organisation>/_apis/distributedtask/pools/<poolid>/jobrequests.

KEDA will interpret this request to find any matching template from the defined parent in the scaling definition, or any agent that can satisfy the demands specified in the scaling definition.

Once it finds it, it will scale the workload that matched the definition and Azure DevOps will assign it to that agent.

However, as this an undocumented API, it possesses some unique quirks when calling it with different query parameters. For example, if the $top query parameter is given, the format of the returned JSON is changed in such a way that it is no longer possible for the scaler to find the matched agents; making it impossible to use with the parent property from the trigger metadata. Therefore making jobsToFetch mutually exclusive with parent in the trigger metadata.

Additionally, the $top query parameter takes precedence over some other parameters like completedRequestCount. If the $top query parameter is given, finished jobs are included in the response even if completedRequestCount=0 is given, although completedRequestCount=0 would indicate that only pending and running jobs should be returned. Thus, jobsToFetch is also mutually exclusive with fetchUnfinishedJobsOnly in the trigger metadata.

Configuring the agent container

Microsoft self-hosted docker agent documentation: https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/docker?view=azure-devops#linux

Please use the script in Step 5 as the entrypoint for your agent container.

You will need to change this section of the shell script so that the agent will terminate and cleanup itself when the job is complete by using the --once switch. The if statement for cleanup is only required if you are using the auto-deployment parent template method.

print_header "4. Running Azure Pipelines agent..."

trap 'cleanup; exit 0' EXIT
trap 'cleanup; exit 130' INT
trap 'cleanup; exit 143' TERM

chmod +x ./run-docker.sh

# To be aware of TERM and INT signals call run.sh
# Running it with the --once flag at the end will shut down the agent after the build is executed
./run-docker.sh "$@" & wait $!

print_header "4. Running Azure Pipelines agent..."

if ! grep -q "template" <<< "$AZP_AGENT_NAME"; then
  echo "Cleanup Traps Enabled"

  trap 'cleanup; exit 0' EXIT
  trap 'cleanup; exit 130' INT
  trap 'cleanup; exit 143' TERM

fi

chmod +x ./run-docker.sh

# To be aware of TERM and INT signals call run.sh
# Running it with the --once flag at the end will shut down the agent after the build is executed
./run-docker.sh "$@" --once & wait $!

Example using Personal Access Token authentication

apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: pipeline-auth
data:
  personalAccessToken: <encoded personalAccessToken>
---
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: pipeline-trigger-auth
  namespace: default
spec:
  secretTargetRef:
    - parameter: personalAccessToken
      name: pipeline-auth
      key: personalAccessToken
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: azure-pipelines-scaledobject
  namespace: default
spec:
  scaleTargetRef:
    name: azdevops-deployment
  minReplicaCount: 1
  maxReplicaCount: 5
  triggers:
  - type: azure-pipelines
    metadata:
      poolID: "1"
      organizationURLFromEnv: "AZP_URL"
      parent: "example-keda-template"
      demands: "maven,docker"
    authenticationRef:
      name: pipeline-trigger-auth

Example using service principal authentication

The following example uses a client secret. Certificate credentials can be configured using the same azureServicePrincipal authentication provider.

apiVersion: v1
kind: Secret
metadata:
  name: azure-pipelines-service-principal
  namespace: default
type: Opaque
data:
  clientSecret: <base64-encoded-client-secret>
---
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: azure-pipelines-service-principal
  namespace: default
spec:
  azureServicePrincipal:
    tenantId: <tenant-id>
    clientId: <client-id>
    clientSecret:
      valueFrom:
        secretKeyRef:
          name: azure-pipelines-service-principal
          key: clientSecret
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: azure-pipelines-scaledobject
  namespace: default
spec:
  scaleTargetRef:
    name: azdevops-deployment
  minReplicaCount: 1
  maxReplicaCount: 5
  triggers:
  - type: azure-pipelines
    metadata:
      poolID: "1"
      organizationURL: "https://dev.azure.com/<organization>"
    authenticationRef:
      name: azure-pipelines-service-principal

Example for Parent Deployment or StatefulSet

apiVersion: apps/v1
kind: Deployment
metadata:
  name: agent
spec:
  replicas: 1
  selector:
    matchLabels:
      app: agent
  spec:
    containers:
      - name: agent
        image: [SAME AS SCALED JOB]
        envFrom:
          - secretRef:
              name: ado-pat-tokens
        env:
          - name: AZP_AGENT_NAME
            value: example-keda-template # Matches Scaled Job Parent

Example for using pod identity authentication

apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: pipeline-trigger-auth
spec:
  podIdentity:
    provider: azure-workload
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: azure-pipelines-scaledobject
  namespace: default
spec:
  scaleTargetRef:
    name: azdevops-deployment
  minReplicaCount: 1
  maxReplicaCount: 5
  triggers:
  - type: azure-pipelines
    metadata:
      poolID: "1"
      organizationURLFromEnv: "AZP_URL"
      parent: "example-keda-template"
      demands: "maven,docker"
    authenticationRef:
      name: pipeline-trigger-auth