Automated Rebuild Triggers

Overview

This system automatically triggers GitHub Actions workflows to rebuild dependent images when their base images are updated.

Architecture

Base Image Update → Kargo Warehouse → Rebuild-Trigger Stage → GitHub API → Workflow Dispatch → Image Build

Flow Example

Base Image Update: Docker Hub publishes node:22-bookworm-slim with a new digest
Kargo Detection: The node-22-bookworm-slim Warehouse detects the new image
Freight Creation: Kargo creates new Freight for the base image
Stage Promotion: The rebuild-trigger-node-22-bookworm-slim Stage is promoted
HTTP Trigger: Stage executes HTTP step to call GitHub API
Workflow Dispatch: GitHub triggers backstage.yml workflow
Image Build: Backstage image is rebuilt with the new base image

Components

1. Warehouses

Base Image Warehouses: Watch upstream images (node, python, etc.)
- node-22-bookworm-slim → watches docker.io/library/node:22-bookworm-slim
- python-3.12-slim → watches docker.io/library/python:3.12-slim
Managed Image Warehouses: Watch your built images
- backstage → watches ghcr.io/craigedmunds/backstage
- uv → watches ghcr.io/craigedmunds/uv

2. Stages

Analysis Stages

analyze-dockerfile-backstage: Analyzes Dockerfile when backstage image updates
analyze-dockerfile-uv: Analyzes Dockerfile when uv image updates

Rebuild-Trigger Stages

rebuild-trigger-backstage: Triggers backstage rebuild when node base image updates
rebuild-trigger-uv: Triggers uv rebuild when python base image updates

3. GitHub Workflows

Each workflow supports workflow_dispatch with inputs:

version_bump: patch/minor/major
triggered_by: Source of the trigger (e.g., “kargo-base-image-update”)
base_image: Name of the base image that triggered the rebuild

Setup

Prerequisites

The system uses the github-credentials secret managed by Kyverno. This secret should already exist in the image-factory-kargo namespace with:

Key: password (contains GitHub Personal Access Token)
Scope required: workflow (Full control of GitHub Actions workflows)

To verify the secret exists:

kubectl get secret github-credentials -n image-factory-kargo

Apply Kargo Manifests

kubectl apply -f cdk8s/image-factory/dist/image-factory.k8s.yaml

Testing

Manual Test: Trigger a Rebuild

You can manually promote a rebuild-trigger stage to test:

# Promote the backstage rebuild-trigger stage
kubectl kargo promote \
    --stage rebuild-trigger-backstage \
    --namespace image-factory-kargo

This will trigger the backstage workflow to rebuild with the latest node base image.

Check Promotion Status

# List all stages
kubectl get stages -n image-factory-kargo
 
# Check specific stage
kubectl get stage rebuild-trigger-backstage -n image-factory-kargo -o yaml
 
# View promotions
kubectl get promotions -n image-factory-kargo

View Logs

# Get recent promotions
kubectl get promotions -n image-factory-kargo --sort-by=.metadata.creationTimestamp | tail -5
 
# View promotion details
kubectl describe promotion <promotion-name> -n image-factory-kargo

Configuration

Adding New Images

Add to image-factory/images.yaml:

- name: myapp
  registry: ghcr.io
  repository: myorg/myapp
  source:
    provider: github
    repo: myorg/myrepo
    branch: main
    dockerfile: apps/myapp/Dockerfile
    workflow: myapp.yml  # GitHub Actions workflow file
  rebuildDelay: 7d
  autoRebuild: true

Regenerate manifests:

cd cdk8s/image-factory
cdk8s synth

Apply:

kubectl apply -f cdk8s/image-factory/dist/image-factory.k8s.yaml

Customizing Rebuild Behavior

Edit the rebuild-trigger stage creation in cdk8s/image-factory/main.py:

# Change version bump type
"version_bump": "minor"  # instead of "patch"
 
# Add custom inputs
"inputs": {
    "version_bump": "patch",
    "triggered_by": "kargo-base-image-update",
    "base_image": base_name,
    "custom_param": "value"
}

Troubleshooting

Workflow Not Triggering

Check secret exists:

kubectl get secret github-token -n image-factory-kargo

Check token has correct scope:
- Token needs workflow scope
- Verify at https://github.com/settings/tokens

Check promotion status:

kubectl get promotions -n image-factory-kargo
kubectl describe promotion <name> -n image-factory-kargo

Check GitHub API response:
- Look at promotion logs for HTTP response codes
- 401 = Authentication failed (bad token)
- 404 = Workflow file not found
- 422 = Invalid inputs

Stage Not Promoting

Check if freight exists:

kubectl get freight -n image-factory-kargo

Check stage configuration:

kubectl get stage rebuild-trigger-node-22-bookworm-slim -n image-factory-kargo -o yaml

Enable auto-promotion (if desired):
- Edit ProjectConfig to add auto-promotion policy

Architecture Decisions

Why HTTP Step Instead of Webhooks?

Simpler: No need to expose Kargo to the internet
Secure: Token stored in Kubernetes secret
Reliable: Direct API call, no webhook delivery issues
Flexible: Easy to customize per-image

Why Separate Rebuild-Trigger Stages?

Clear separation: Analysis vs. rebuild triggering
Independent control: Can disable rebuilds without affecting analysis
Visibility: Easy to see which base image triggered which rebuild
Flexibility: Different rebuild strategies per base image

Future Enhancements

Add retry logic for failed HTTP calls
Add notifications (Slack, email) when rebuilds are triggered
Support for GitLab CI/CD triggers
Configurable rebuild delays (wait N hours after base update)
Smart rebuild scheduling (avoid peak hours)

Techcle Wiki

Explorer

REBUILD TRIGGERS