Spec 0006: K8S Lab - Authenticated External Ingress

Overview

Implement hostname-based authentication at the Traefik edge gateway such that:

*.lab.ctoaas.co requires Google OAuth authentication (opt-in via Helm values)
*.lab.local.ctoaas.co does not require authentication (always accessible)

Public exposure is opt-in: By default, services are only accessible via internal hostnames (*.lab.local.ctoaas.co). To expose a service publicly at *.lab.ctoaas.co, set accessPattern: public in the ingress Helm values.

Goals

Explicit opt-in - Only ingresses with accessPattern: public are publicly accessible
Central enforcement - Authentication applied via Traefik Middleware CRD
GitOps compliant - All configuration declared in Helm values, fully reconciled by ArgoCD
Single entrypoint - All traffic uses existing websecure entryPoint on port 443

Non-Goals

Per-service authentication configuration
Switching from standard Kind: Ingress to Traefik IngressRoute CRDs
Separate entryPoints or ports for authenticated vs non-authenticated traffic

Technical Design

Architecture

The implementation uses a two-ingress pattern for public services:

Client Request (HTTPS/443)
         │
         ▼
┌─────────────────────────────────────────────────┐
│              Traefik (websecure)                │
│                                                 │
│  ┌─────────────────────────────────────────┐   │
│  │ Middleware: google-auth (CRD)           │   │
│  │ Type: forwardAuth                       │   │
│  │ Address: http://oauth2-proxy/oauth2/    │   │
│  │ Headers: X-Auth-Request-User, etc.      │   │
│  └─────────────────────────────────────────┘   │
│                                                 │
└─────────────────────────────────────────────────┘
         │
         ▼
┌─────────────────────────────────────────────────┐
│         Kubernetes Ingress Resources            │
│                                                 │
│  Internal Ingress (e.g., code-server):         │
│  - Host: code-server.lab.local.ctoaas.co       │
│  - No middleware (internal access)             │
│                                                 │
│  Public Ingress (e.g., code-server-public):    │
│  - Host: code-server.lab.ctoaas.co             │
│  - Middleware: traefik-google-auth@kubernetescrd│
│                                                 │
└─────────────────────────────────────────────────┘

Two-Ingress Pattern

For services with accessPattern: public, the Helm chart creates two ingresses:

Internal ingress (e.g., code-server)
- Hostname: *.lab.local.ctoaas.co
- No auth middleware
- For internal/cluster access
Public ingress (e.g., code-server-public)
- Hostname: *.lab.ctoaas.co
- Has auth middleware annotation
- For external access with Google OAuth

Auth Bypass with skipAuth

The oauth2-proxy ingress needs public access but must NOT have auth middleware (would cause redirect loop). Use skipAuth: true:

ingress:
  accessPattern: public
  skipAuth: true  # No middleware on -public ingress

Components

1. traefik-ingress Helm Chart

Located in libs/workspace-shared/helm/traefik-ingress/, this chart:

Creates internal ingresses for all services
Creates -public ingresses for accessPattern: public services
Applies auth middleware annotation to public ingresses (unless skipAuth: true)

2. google-auth Middleware CRD

Located in traefik/google-auth-middleware.yaml:

apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: google-auth
  namespace: traefik
spec:
  forwardAuth:
    address: http://oauth2-proxy.oauth2-proxy.svc.cluster.local/oauth2/
    trustForwardHeader: true
    authResponseHeaders:
      - X-Auth-Request-User
      - X-Auth-Request-Email
      - X-Auth-Request-Access-Token
      - Authorization

3. oauth2-proxy

Deployed in components/oauth2-proxy/:

Google OAuth configuration
Cookie-based session storage
Email allowlist via ConfigMap
Credentials via ClusterExternalSecret from central-secret-store

Configuration Example

In components/ingress/kustomization.yaml:

helmCharts:
  - name: traefik-ingress
    valuesInline:
      domains:
        localDomainSuffix: lab.local.ctoaas.co
        publicDomainSuffix: lab.ctoaas.co
 
      auth:
        middleware: traefik-google-auth@kubernetescrd
 
      ingresses:
        # Internal-only service
        - service:
            name: headlamp
            namespace: headlamp
          ingress:
            accessPattern: internal  # Only internal ingress created
 
        # Public service with auth
        - service:
            name: code-server
            namespace: code-server
          ingress:
            accessPattern: public  # Creates both internal and -public ingresses
 
        # Public service without auth (oauth callback)
        - service:
            name: oauth2-proxy
            namespace: oauth2-proxy
          ingress:
            accessPattern: public
            skipAuth: true  # -public ingress has no middleware

Validation Criteria

Unit Tests (MR Checks)

Run with task test:mr in components/ingress/:

Test	Expected Result
`code-server-public` ingress	Has `traefik-google-auth@kubernetescrd` middleware
`code-server` ingress	No middleware annotation
`oauth2-proxy-public` ingress	No middleware annotation (skipAuth)
Public ingresses	Only have `*.lab.ctoaas.co` hostname
Internal ingresses	Only have `*.lab.local.ctoaas.co` hostname

Acceptance Tests (Cluster)

Run with task test:acceptance in repo root (requires cluster access):

Test	Expected Result
Visit `https://code-server.lab.ctoaas.co`	Redirects to Google login
Visit `https://code-server.lab.local.ctoaas.co`	Direct access, no auth
Visit `https://auth.lab.ctoaas.co`	oauth2-proxy accessible (no auth loop)
Traefik dashboard	Shows `google-auth` middleware

File Changes

New/Modified Files

libs/workspace-shared/helm/traefik-ingress/
├── Chart.yaml
├── values.yaml                    # Added auth.middleware config
└── templates/ingress.yaml         # Two-ingress pattern for public access

components/ingress/
├── kustomization.yaml             # Ingress definitions with accessPattern
├── Taskfile.yaml                  # test:mr task (runs test:deps + test:unit)
└── tests/unit/
    ├── requirements.txt           # pytest, pyyaml
    └── test_ingress_manifests.py  # Spec 0006 validation tests

components/oauth2-proxy/           # oauth2-proxy deployment
traefik/google-auth-middleware.yaml # Middleware CRD

Note: General MR checks infrastructure (GitHub Actions workflow, discovery script) is documented separately at the workspace level.

Dependencies

Google OAuth application credentials (stored in central-secret-store)
DNS for *.lab.ctoaas.co and *.lab.local.ctoaas.co
TLS certificates (cert-manager with letsencrypt-prod)
External Secrets Operator
ClusterSecretStore central-secret-store

Techcle Wiki

Explorer

Spec