Image Factory End-to-End Workflow

This document describes the complete end-to-end workflow for the Image Factory system, which automates container image building, promotion, and deployment using Kargo and ArgoCD.

Repository Structure

The Image Factory is split across three repositories:

image-factory - Code repository
- CDK8s application for generating Kargo resources
- Python tools for image analysis and discovery
- Tests (unit, integration, acceptance)
- GitHub Actions workflows
image-factory-state - State repository
- images.yaml - Image enrollment registry (source of truth)
- base-images/ - State files for base container images
- images/ - State files for application images
- dist/ - Generated Kargo manifests (CDK8s output)
argocd-eda - Platform repository
- ArgoCD application that deploys Image Factory
- Backstage integration plugins

Architecture

┌─────────────────────────────────────────────────────────────────────┐
│                         Developer Actions                            │
├─────────────────────────────────────────────────────────────────────┤
│ 1. Update image-factory code (CDK8s, tools)                         │
│ 2. Update image-factory-state config (images.yaml, state files)     │
└────────────────┬────────────────────────────────────────────────────┘
                 │
                 ▼
┌─────────────────────────────────────────────────────────────────────┐
│                      GitHub Actions Workflows                        │
├─────────────────────────────────────────────────────────────────────┤
│ image-factory repo:                                                  │
│   - synth.yml: CDK8s synthesis (on code changes)                    │
│   - test.yml: Run tests                                             │
│                                                                      │
│ Process:                                                             │
│   1. Checkout both repos (code + state)                             │
│   2. Run CDK8s synthesis                                            │
│   3. Commit generated manifests to image-factory-state/dist/        │
└────────────────┬────────────────────────────────────────────────────┘
                 │
                 ▼
┌─────────────────────────────────────────────────────────────────────┐
│                          ArgoCD Sync                                 │
├─────────────────────────────────────────────────────────────────────┤
│ ArgoCD watches: image-factory-state repo, path: dist/               │
│                                                                      │
│ When dist/ changes:                                                  │
│   - ArgoCD detects change                                           │
│   - Syncs Kargo resources to cluster                                │
│   - Creates/updates Warehouses, Stages, AnalysisTemplates           │
└────────────────┬────────────────────────────────────────────────────┘
                 │
                 ▼
┌─────────────────────────────────────────────────────────────────────┐
│                        Kargo Orchestration                           │
├─────────────────────────────────────────────────────────────────────┤
│ Warehouses:                                                          │
│   - Track base images (node, python, etc.)                          │
│   - Track application images (backstage, uv, etc.)                  │
│   - Monitor Git repos for source changes                            │
│                                                                      │
│ Stages (Rebuild Triggers):                                          │
│   - Detect base image updates                                       │
│   - Trigger GitHub workflow for dependent app builds                │
│   - Respect rebuildDelay configuration                              │
│                                                                      │
│ Stages (Analysis):                                                   │
│   - Run Dockerfile analysis on new images                           │
│   - Validate image contents                                         │
│   - Promote images through environments                             │
└─────────────────────────────────────────────────────────────────────┘

End-to-End Workflows

1. Adding a New Image to Track

Steps:

Edit image-factory-state/images.yaml:

- name: my-new-app
  registry: ghcr.io
  repository: craigedmunds/my-new-app
  source:
    provider: github
    repo: craigedmunds/argocd-eda
    branch: main
    dockerfile: apps/my-new-app/Dockerfile
    workflow: my-new-app.yml
  rebuildDelay: 7d
  autoRebuild: true

Commit and push to image-factory-state repo
Automated process:
- image-factory workflows detect state change
- CDK8s synthesizes new Kargo resources
- New manifests committed to image-factory-state/dist/
- ArgoCD detects change and syncs to cluster
- Kargo starts tracking the new image

2. Updating Image Factory Code

Steps:

Make changes to image-factory repo (CDK8s code, tools, etc.)
Create PR in image-factory repo
Automated process:
- Tests run via test.yml workflow
- On merge to main:
  - synth.yml workflow runs
  - Checks out both repos
  - Runs CDK8s synthesis
  - Commits updated manifests to image-factory-state
- ArgoCD detects change and syncs

3. Base Image Update Triggering Rebuild

Automatic process:

Kargo Warehouse detects new base image digest (e.g., node:22-bookworm-slim)
Kargo Stage (rebuild trigger) evaluates:
- Which application images depend on this base image
- Whether rebuildDelay has passed since last rebuild
- Whether autoRebuild is enabled
If criteria met, Kargo triggers GitHub workflow dispatch:
- Workflow builds new image
- Pushes to GHCR
- Tags with semantic version
Kargo analysis stage runs:
- Validates Dockerfile
- Runs security scans
- Promotes image through environments

4. Manual Image Rebuild

Steps:

Navigate to GitHub Actions in source repo
Trigger workflow manually via workflow_dispatch
Kargo detects new image and runs analysis

Configuration Files

images.yaml (Source of Truth)

Location: image-factory-state/images.yaml

- name: backstage
  registry: ghcr.io
  repository: craigedmunds/backstage
  source:
    provider: github
    repo: craigedmunds/argocd-eda
    branch: main
    dockerfile: backstage/app/packages/backend/Dockerfile
    workflow: backstage.yml
  rebuildDelay: 7d
  autoRebuild: true

State Files

Generated and managed by image-factory tools:

image-factory-state/images/{name}.yaml - Application image state
image-factory-state/base-images/{name}.yaml - Base image state

State files contain:

Enrollment metadata
Current version/digest
Dependency information
Rebuild state
Warehouse configuration

GitHub Actions Workflows

image-factory/synth.yml

Trigger: PR to image-factory repo with CDK8s changes

Process:

Checkout image-factory (code)
Checkout image-factory-state (config)
Setup CDK8s environment
Run synthesis: python main.py
Check for changes in dist/
Commit changes to image-factory-state repo
Comment on PR

Environment Variables:

IMAGE_FACTORY_STATE_DIR: Path to state repo (default: ../../image-factory-state)

image-factory/test.yml

Trigger: PR to image-factory repo

Process:

Run unit tests
Run integration tests
Run acceptance tests

ArgoCD Application

Location: argocd-eda/platform/kustomize/seed/image-factory/image-factory-app.yaml

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: image-factory
  namespace: argocd
  labels:
    repo: image-factory-state
    capability: image-factory
spec:
  project: sdlc
  source:
    repoURL: https://github.com/craigedmunds/image-factory-state.git
    path: dist
    targetRevision: main
  destination:
    server: https://kubernetes.default.svc
    namespace: image-factory
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Local Development

Testing CDK8s Synthesis Locally

# Clone both repos
git clone git@github.com:craigedmunds/image-factory.git
git clone git@github.com:craigedmunds/image-factory-state.git
 
cd image-factory
 
# Setup environment
task cdk8s:setup
 
# Run synthesis
cd cdk8s
export IMAGE_FACTORY_STATE_DIR=../../image-factory-state
.venv/bin/python main.py
 
# Check generated output
ls -la ../../image-factory-state/dist/

Running Tests

cd image-factory
 
# Run all tests
task test:all
 
# Run specific test suites
task test:unit
task test:integration
task test:acceptance

Troubleshooting

Synthesis Not Updating dist/

Check:

Is IMAGE_FACTORY_STATE_DIR set correctly?
Are state files valid YAML?
Check synthesis logs for errors

Debug:

cd image-factory/cdk8s
export IMAGE_FACTORY_STATE_DIR=../../image-factory-state
.venv/bin/python main.py

ArgoCD Not Syncing

Check:

ArgoCD application source points to image-factory-state repo
Path is set to dist/
Check ArgoCD sync status: kubectl get app image-factory -n argocd

Debug:

# Check ArgoCD application
kubectl describe app image-factory -n argocd
 
# Check sync status
argocd app get image-factory
 
# Force sync
argocd app sync image-factory

Kargo Not Detecting Images

Check:

Warehouses created correctly: kubectl get warehouses -n image-factory
Image registry credentials configured
Check Warehouse status for errors

Debug:

# List warehouses
kubectl get warehouses -n image-factory
 
# Check warehouse details
kubectl describe warehouse <warehouse-name> -n image-factory
 
# Check Kargo controller logs
kubectl logs -n kargo -l app.kubernetes.io/component=controller

GitHub Actions Workflow Not Triggering

Check:

Workflow file in correct location
Trigger paths match changed files
Required secrets configured (GITHUB_TOKEN)

Debug:

# View workflow runs
gh run list --workflow=synth.yml
 
# View specific run
gh run view <run-id> --log

Best Practices

1. State File Management

Never manually edit state files - use images.yaml instead
Commit state files to git
Review generated manifests before deploying

2. Version Control

Use semantic versioning for application images
Tag releases in GitHub
Use git commit SHA for development builds

3. Security

Scan images before deployment (via Kargo analysis)
Use private registries for proprietary images
Rotate registry credentials regularly

4. Rebuild Configuration

Set appropriate rebuildDelay (7d for apps, 30d for infrastructure)
Use autoRebuild: false for critical production images
Test rebuild triggers in non-production environments first

5. Monitoring

Watch Kargo stage status
Monitor ArgoCD sync health
Set up alerts for build/deploy failures

Docker Image Build Workflow - Standard Docker build patterns
ArgoCD Development Workflow - ArgoCD best practices
Image Factory Project - Project documentation
image-factory README - Code repository docs
image-factory-state README - State repository docs

Techcle Wiki

Explorer

Image Factory Workflow

Image Factory End-to-End Workflow

Repository Structure

Architecture

End-to-End Workflows

1. Adding a New Image to Track

2. Updating Image Factory Code

3. Base Image Update Triggering Rebuild

4. Manual Image Rebuild

Configuration Files

images.yaml (Source of Truth)

State Files

GitHub Actions Workflows

image-factory/synth.yml

image-factory/test.yml

ArgoCD Application

Local Development

Testing CDK8s Synthesis Locally

Running Tests

Troubleshooting

Synthesis Not Updating dist/

ArgoCD Not Syncing

Kargo Not Detecting Images

GitHub Actions Workflow Not Triggering

Best Practices

1. State File Management

2. Version Control

3. Security

4. Rebuild Configuration

5. Monitoring

Graph View

Table of Contents

Techcle Wiki

Explorer

Image Factory Workflow

Image Factory End-to-End Workflow

Repository Structure

Architecture

End-to-End Workflows

1. Adding a New Image to Track

2. Updating Image Factory Code

3. Base Image Update Triggering Rebuild

4. Manual Image Rebuild

Configuration Files

images.yaml (Source of Truth)

State Files

GitHub Actions Workflows

image-factory/synth.yml

image-factory/test.yml

ArgoCD Application

Local Development

Testing CDK8s Synthesis Locally

Running Tests

Troubleshooting

Synthesis Not Updating dist/

ArgoCD Not Syncing

Kargo Not Detecting Images

GitHub Actions Workflow Not Triggering

Best Practices

1. State File Management

2. Version Control

3. Security

4. Rebuild Configuration

5. Monitoring

Related Documentation

Graph View

Table of Contents