Dynamic pipelines

Premium-only feature

Dynamic pipelines is a Premium feature for Bitbucket Cloud. Read more about Bitbucket Premium.

Overview

Dynamic Pipelines allow you to programmatically generate the pipeline YAML at runtime using a script. Instead of a static bitbucket-pipelines.yml, a generator script produces the pipeline configuration on the fly, which is then uploaded and executed.

Consider simpler alternatives first

Dynamic Pipelines add significant complexity. Before using them, confirm you cannot solve your problem with:

YAML Sharing — Eliminate cross-repo duplication
YAML Templating — Parameterize shared templates
Dynamic Conditions — Conditionally skip steps
Parent/Child Pipelines — Orchestrate multiple services

Dynamic Pipelines are appropriate for: test matrices derived from code analysis, monorepos with 100+ services where even conditional steps create overhead, or pipelines driven by external data (e.g., per-customer configurations).

How It Works

A step runs a generator script that produces a YAML or JSON pipeline definition
The step pipes the output to a file
The step uses the atlassian/bitbucket-upload-generated-pipeline pipe to upload and execute the generated pipeline

The `bitbucket-upload-generated-pipeline` Pipe

- pipe: atlassian/bitbucket-upload-generated-pipeline:1.0.0
  variables:
    GENERATED_PIPELINE_FILE: generated-pipeline.yml

Variables

Variable	Required	Description
`GENERATED_PIPELINE_FILE`	✅	Path to the generated YAML or JSON file

Basic Pipeline Configuration

# bitbucket-pipelines.yml
image: node:20
pipelines:
  default:
    - step:
        name: Generate Pipeline
        script:
          - node generate-pipeline.js > generated-pipeline.yml
          - pipe: atlassian/bitbucket-upload-generated-pipeline:1.0.0
            variables:
              GENERATED_PIPELINE_FILE: generated-pipeline.yml

Generator Script Format

The generator outputs either YAML or JSON representing a valid Bitbucket Pipelines configuration.

JSON Output (JavaScript)

// generate-pipeline.js
function generatePipeline() {
  return {
    image: 'node:20',
    pipelines: {
      default: [
        {
          step: {
            name: 'Build',
            script: ['npm install', 'npm test', 'npm run build']
          }
        }
      ]
    }
  };
}
console.log(JSON.stringify(generatePipeline(), null, 2));

YAML Output

# generate-pipeline.py
import yaml
import sys
pipeline = {
    'image': 'node:20',
    'pipelines': {
        'default': [
            {
                'step': {
                    'name': 'Build',
                    'script': ['npm install', 'npm test']
                }
            }
        ]
    }
}
print(yaml.dump(pipeline))

Generated Pipeline Schema

The generated file must conform to standard Bitbucket Pipelines YAML structure:

{
  "image": "node:20",
  "options": {
    "max-time": 60
  },
  "pipelines": {
    "default": [
      {
        "step": {
          "name": "Step Name",
          "script": ["command1", "command2"],
          "caches": ["node"],
          "artifacts": ["dist/**"]
        }
      }
    ]
  }
}

All standard pipeline features (parallel steps, stages, caches, artifacts, services) are supported in generated pipelines.

Common Use Case: Monorepo with Changed-Service Detection

// generate-monorepo-pipeline.js
const { execSync } = require('child_process');
// Detect which services changed
const changedFiles = execSync('git diff --name-only HEAD~1').toString().trim();
const changedServices = new Set();
changedFiles.split('\n').forEach(file => {
  const match = file.match(/^services\/([^\/]+)\//);
  if (match) changedServices.add(match[1]);
});
// Generate a step only for changed services
const steps = Array.from(changedServices).map(service => ({
  step: {
    name: `Build ${service}`,
    script: [
      `cd services/${service}`,
      'npm ci',
      'npm test',
      'npm run build'
    ]
  }
}));
// Fallback if nothing changed
if (steps.length === 0) {
  steps.push({
    step: {
      name: 'No services changed',
      script: ['echo "Nothing to build"']
    }
  });
}
console.log(JSON.stringify({ pipelines: { default: steps } }, null, 2));

Common Use Case: Code-Analysis-Based Test Matrix

// generate-test-matrix.js
const fs = require('fs');
const pkg = JSON.parse(fs.readFileSync('package.json', 'utf8'));
const deps = { ...pkg.dependencies, ...pkg.devDependencies };
const steps = [];
if (deps['react'] || deps['vue'] || deps['angular']) {
  steps.push({
    step: { name: 'Unit Tests', script: ['npm run test:unit'] }
  });
}
if (deps['playwright'] || deps['cypress']) {
  steps.push({
    step: { name: 'E2E Tests', script: ['npm run test:e2e'] }
  });
}
if (deps['jest'] || deps['vitest']) {
  steps.push({
    step: { name: 'Coverage Report', script: ['npm run test:coverage'] }
  });
}
console.log(JSON.stringify({
  image: 'node:20',
  pipelines: { default: steps }
}, null, 2));

Debugging Generated Pipelines

Since the generated YAML is not in source control, debugging requires logging:

- step:
    name: Generate Pipeline
    script:
      - node generate-pipeline.js > generated-pipeline.yml
      - echo "=== Generated Pipeline ===" && cat generated-pipeline.yml  # Log output
      - pipe: atlassian/bitbucket-upload-generated-pipeline:1.0.0
        variables:
          GENERATED_PIPELINE_FILE: generated-pipeline.yml

Use console.error() in your generator for debug info (appears in logs without affecting YAML output):

console.error('Changed services:', Array.from(changedServices));
console.log(JSON.stringify(pipeline));

Error Handling

Always include a fallback for generator failures:

try {
  const pipeline = generatePipeline();
  console.log(JSON.stringify(pipeline));
} catch (error) {
  console.error('Generator failed:', error.message);
  // Emit a minimal fallback pipeline
  console.log(JSON.stringify({
    pipelines: {
      default: [{
        step: {
          name: 'Pipeline generation failed',
          script: ['echo "Generator error: see logs above"', 'exit 1']
        }
      }]
    }
  }));
}

Constraints

Constraint	Value
Generated pipeline size	1 MB max
Steps in generated pipeline	100 max
Supported output formats	JSON or YAML
Generator script language	Any (Node.js, Python, Bash, etc.)
Access to Bitbucket variables	Available as environment variables in generator step
Generated pipelines can trigger children	✅ Yes

Testing Your Generator Locally

# Run the generator to inspect output
node generate-pipeline.js
# Validate the output structure
node generate-pipeline.js | python3 -c "import sys, json; json.load(sys.stdin); print('Valid JSON')"
# Test with simulated environment variables
BITBUCKET_BRANCH=main node generate-pipeline.js

Best Practices

Keep generation logic simple — Complex generators are hard to debug and maintain
Log what was generated — Always cat the generated file in the pipeline step
Include a fallback — Generator failures should produce a minimal pipeline, not silently fail
Test the generator locally — Run the script in your dev environment before committing
Version the generator — Tag generator releases so you can roll back if it breaks

Dynamic conditions reference — Simpler alternative for conditional logic
Parent/Child pipelines reference — Orchestrate multiple pipelines without code generation
Step-by-step tutorial
Dynamic pipelines Forge module reference

Was this helpful?

It wasn't accurateIt wasn't clearIt wasn't relevant

Still need help?

The Atlassian Community is here for you.

Ask the Community