Review 0006: K8S Lab - Authenticated External Ingress
Implementation Summary
This review documents the implementation of hostname-based authentication at the Traefik edge gateway, as specified in codev/specs/0006-authenticated-external-ingress.md.
Requirements Verification
Functional Requirements
| ID | Requirement | Status | Notes |
|---|---|---|---|
| FR-1 | Single EntryPoint | ✅ Pass | All traffic uses websecure entryPoint on port 443 |
| FR-2 | Hostname-Based Policy | ✅ Pass | *.lab.ctoaas.co requires OAuth, *.lab.local.ctoaas.co bypasses |
| FR-3 | Centralized Enforcement | ✅ Pass | Auth at Traefik via ForwardAuth middleware |
| FR-4 | OAuth Integration | ✅ Pass | oauth2-proxy handles authentication |
| FR-5 | GitOps Compliance | ✅ Pass | All config in Helm values/Kustomize |
Technical Requirements
| ID | Requirement | Status | Notes |
|---|---|---|---|
| TR-1 | Providers | ✅ Pass | kubernetesCRD and kubernetesIngress enabled |
| TR-2 | Auth middleware | ✅ Pass | Using CRD-based Middleware (simpler than file provider) |
| TR-3 | Auth routing | ✅ Pass | Middleware applied via ingress annotation |
| TR-4 | Middleware config | ✅ Pass | ForwardAuth with header forwarding |
| TR-5 | oauth2-proxy | ✅ Pass | Deployed via Helm chart |
| TR-6 | Secret Management | ✅ Pass | ClusterExternalSecret syncs from central-secret-store |
| TR-7 | Email Allowlist | ✅ Pass | ConfigMap mounted to oauth2-proxy |
Implementation Deviations from Plan
1. CRD Middleware vs File Provider
Plan: Use Traefik’s file provider with dynamic configuration for routers and middleware.
Implementation: Use Kubernetes CRD-based Middleware resource in traefik namespace.
Rationale: CRD-based approach is:
- Simpler to deploy (no ConfigMap mounting needed)
- More Kubernetes-native (kubectl get middleware)
- Easier to debug (visible in Traefik dashboard)
- Follows existing patterns in the codebase
2. ForwardAuth Endpoint Change
Plan: Use /oauth2/auth endpoint for ForwardAuth.
Implementation: Use /oauth2/ endpoint instead.
Rationale: The /oauth2/auth endpoint returns 401 for unauthenticated requests, requiring a separate Errors middleware to handle the redirect. The /oauth2/ endpoint returns 302 directly, which is the expected behavior for ForwardAuth. This is the standard pattern documented by oauth2-proxy.
3. Secret Naming
Plan: Secret named google-oauth.
Implementation: Secret named gcp-credentials for broader GCP credential storage.
Rationale: More descriptive name that allows for future expansion of GCP credentials.
4. Ingress Management via Helm
Plan: Individual ingress resources with annotations.
Implementation: Centralized ingress management via components/ingress Helm chart.
Rationale:
- Single source of truth for all ingress configurations
- Consistent patterns across services
- Easier to apply auth middleware to all public ingresses
5. Two-Ingress Pattern
Plan: Single ingress with conditional middleware based on hostname match.
Implementation: Two separate ingresses for public services:
- Internal ingress (e.g.,
code-server): Only*.lab.local.ctoaas.cohostname, no middleware - Public ingress (e.g.,
code-server-public): Only*.lab.ctoaas.cohostname, with auth middleware
Rationale:
- Cleaner separation of concerns
- Easier to reason about which ingress has auth
- No hostname-matching complexity in middleware
skipAuth: trueoption for services like oauth2-proxy that need public access without auth
Test Results
Unit Tests (MR Checks)
All 12 unit tests pass in components/ingress/tests/unit/:
✓ test_produces_output - Kustomize generates valid output
✓ test_produces_ingresses - Multiple ingress resources generated
✓ test_code_server_public_exists - code-server-public ingress exists
✓ test_code_server_public_has_auth_middleware - code-server-public has google-auth
✓ test_codev_public_exists - codev-public ingress exists
✓ test_codev_public_has_auth_middleware - codev-public has google-auth
✓ test_public_ingresses_use_correct_middleware_format - Correct middleware annotation format
✓ test_public_ingresses_have_public_hostname - Public ingresses use *.lab.ctoaas.co
✓ test_oauth2_proxy_public_exists - oauth2-proxy-public ingress exists
✓ test_oauth2_proxy_public_no_auth_middleware - oauth2-proxy has no auth (skipAuth)
✓ test_internal_ingresses_no_auth - Internal ingresses have no middleware
✓ test_internal_ingresses_have_internal_hostname - Internal ingresses use *.lab.local.ctoaas.co
Acceptance Tests (Cluster)
All 8 acceptance tests pass:
✓ oauth2-proxy pod is running
✓ google-auth middleware exists in traefik namespace
✓ code-server.lab.ctoaas.co returns 302 redirect to Google OAuth
✓ codev.lab.ctoaas.co returns 302 redirect to Google OAuth
✓ oauth2-proxy health check (/ping) returns 200
✓ code-server-public ingress has google-auth middleware
✓ codev-public ingress has google-auth middleware
✓ WebSocket upgrade request handled correctly (status: 302)
Key Files Changed
New Files
| File | Purpose |
|---|---|
repos/k8s-lab/components/oauth2-proxy/ | oauth2-proxy deployment |
repos/k8s-lab/traefik/google-auth-middleware.yaml | ForwardAuth middleware |
repos/k8s-lab/components/central-secret-store/external-secrets/gcp-credentials.yaml | GCP credentials sync |
repos/k8s-lab/tests/authenticated-ingress-test.sh | Acceptance tests |
repos/k8s-lab/components/ingress/Taskfile.yaml | Component tasks with test:mr |
repos/k8s-lab/components/ingress/tests/unit/ | Unit tests for Spec 0006 validation |
repos/k8s-lab/scripts/discover-mr-targets.py | MR target discovery for CI |
repos/k8s-lab/.github/workflows/mr-checks.yaml | GitHub Actions MR validation |
repos/k8s-lab/tests/validate-kustomizations.sh | Local MR validation script |
Modified Files
| File | Changes |
|---|---|
repos/k8s-lab/traefik/values.yaml | Added allowCrossNamespace: true |
repos/k8s-lab/traefik/kustomization.yaml | Added middleware resource |
repos/k8s-lab/components/ingress/kustomization.yaml | Added auth middleware config |
repos/k8s-lab/components/central-secret-store/kustomization.yaml | Added gcp-credentials |
repos/k8s-lab/Taskfile.yaml | Added test:acceptance tasks |
Lessons Learned
1. oauth2-proxy Endpoints Matter
The difference between /oauth2/auth and /oauth2/ endpoints is significant:
/oauth2/auth: Returns 401 for unauthenticated (requires additional error handling)/oauth2/: Returns 302 for unauthenticated (works directly with ForwardAuth)
This is a common source of confusion when configuring oauth2-proxy with Traefik.
2. Cross-Namespace Middleware References
Traefik’s CRD provider requires explicit allowCrossNamespace: true to reference services in other namespaces from ForwardAuth middleware.
3. WebSocket Proxies
When using external nginx proxies (e.g., Nginx Proxy Manager) in front of the cluster, WebSocket support must be explicitly enabled at each proxy layer.
4. Ingress Management Centralization
Managing ingresses through a centralized Helm chart (components/ingress) provides better consistency than individual ingress resources scattered across components.
Security Considerations
- OAuth credentials: Stored in central-secret-store, synced via External Secrets Operator
- Cookie security: Secure cookies enabled, domain-scoped to
.lab.ctoaas.co - Email allowlist: Only approved emails can access protected services
- No secrets in Git: All secrets managed via External Secrets
Future Improvements
- Add rate limiting middleware for auth endpoints
- Consider Redis for session storage (if scaling oauth2-proxy)
- Add monitoring/alerting for auth failures
- Document the email allowlist management process
Verdict
APPROVE - Implementation meets all spec requirements with appropriate deviations documented. All acceptance tests pass. The implementation is GitOps-compliant and follows Kubernetes best practices.