Techcle Wiki

Plan

5 min read

Plan 0006: K8S Lab - Authenticated External Ingress

Overview

Implementation plan for hostname-based edge authentication with Traefik. This plan follows the spec at codev/specs/0006-authenticated-external-ingress.md.

Prerequisites

  • Google OAuth application created in Google Cloud Console
    • Authorized redirect URI: https://auth.lab.ctoaas.co/oauth2/callback
    • Note the Client ID and Client Secret

Phase 1: Secret Infrastructure

Task 1.1: Create Source Secret (Manual)

# Generate a cookie secret
COOKIE_SECRET=$(openssl rand -base64 32 | tr -d '\n')
 
# Create the source secret in central-secret-store namespace
kubectl create secret generic google-oauth -n central-secret-store \
  --from-literal=client-id=<GOOGLE_CLIENT_ID> \
  --from-literal=client-secret=<GOOGLE_CLIENT_SECRET> \
  --from-literal=cookie-secret=$COOKIE_SECRET

Task 1.2: Create ClusterExternalSecret

File: repos/k8s-lab/components/central-secret-store/external-secrets/google-oauth.yaml

apiVersion: external-secrets.io/v1beta1
kind: ClusterExternalSecret
metadata:
  name: google-oauth-credentials
spec:
  refreshTime: 5m
 
  namespaceSelector:
    matchLabels:
      secrets/google-oauth-credentials: "true"
 
  externalSecretSpec:
    secretStoreRef:
      name: central-secret-store
      kind: ClusterSecretStore
 
    target:
      name: oauth2-proxy-secrets
      creationPolicy: Owner
      template:
        metadata:
          labels:
            managed-by: external-secrets
            source: central-secret-store
            secret-type: google-oauth-credentials
 
    data:
    - secretKey: client-id
      remoteRef:
        key: google-oauth
        property: client-id
 
    - secretKey: client-secret
      remoteRef:
        key: google-oauth
        property: client-secret
 
    - secretKey: cookie-secret
      remoteRef:
        key: google-oauth
        property: cookie-secret

Task 1.3: Update central-secret-store kustomization

File: repos/k8s-lab/components/central-secret-store/kustomization.yaml

Add to resources:

  - external-secrets/google-oauth.yaml

Phase 2: oauth2-proxy Deployment

Task 2.1: Create oauth2-proxy Directory Structure

repos/k8s-lab/oauth2-proxy/
├── kustomization.yaml
├── oauth2-proxy-namespace.yaml
├── values.yaml
└── allowed-emails.yaml

Task 2.2: Create Namespace

File: repos/k8s-lab/oauth2-proxy/oauth2-proxy-namespace.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: oauth2-proxy
  labels:
    secrets/google-oauth-credentials: "true"

Task 2.3: Create Allowed Emails ConfigMap

File: repos/k8s-lab/oauth2-proxy/allowed-emails.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: oauth2-proxy-allowed-emails
  namespace: oauth2-proxy
data:
  allowed-emails.txt: |
    # Add allowed email addresses, one per line
    # Example:
    # user@example.com

Task 2.4: Create Helm Values

File: repos/k8s-lab/oauth2-proxy/values.yaml

# oauth2-proxy configuration for Google OAuth
 
image:
  repository: quay.io/oauth2-proxy/oauth2-proxy
  tag: "v7.6.0"
 
config:
  clientID: ""      # Set via secret
  clientSecret: ""  # Set via secret
  cookieSecret: ""  # Set via secret
 
  configFile: |-
    provider = "google"
    email_domains = ["*"]
    authenticated_emails_file = "/etc/oauth2-proxy/allowed-emails.txt"
    upstreams = ["static://200"]
    http_address = "0.0.0.0:4180"
    cookie_secure = true
    cookie_domains = [".lab.ctoaas.co"]
    whitelist_domains = [".lab.ctoaas.co"]
    set_xauthrequest = true
    set_authorization_header = true
    pass_access_token = true
    skip_provider_button = true
 
extraArgs:
  - --reverse-proxy=true
 
existingSecret: oauth2-proxy-secrets
 
extraVolumes:
  - name: allowed-emails
    configMap:
      name: oauth2-proxy-allowed-emails
 
extraVolumeMounts:
  - name: allowed-emails
    mountPath: /etc/oauth2-proxy
    readOnly: true
 
ingress:
  enabled: true
  className: ""
  annotations:
    traefik.ingress.kubernetes.io/router.entrypoints: websecure
    traefik.ingress.kubernetes.io/router.tls: "true"
  hosts:
    - host: auth.lab.ctoaas.co
      paths:
        - path: /
          pathType: Prefix
  tls:
    - hosts:
        - auth.lab.ctoaas.co
 
tolerations:
  - key: node-role.kubernetes.io/control-plane
    operator: Exists
    effect: NoSchedule

Task 2.5: Create Kustomization

File: repos/k8s-lab/oauth2-proxy/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
 
namespace: oauth2-proxy
 
resources:
  - oauth2-proxy-namespace.yaml
  - allowed-emails.yaml
 
helmCharts:
  - name: oauth2-proxy
    repo: https://oauth2-proxy.github.io/manifests
    version: 7.7.1
    releaseName: oauth2-proxy
    namespace: oauth2-proxy
    valuesFile: values.yaml

Phase 3: Traefik File Provider Configuration

Task 3.1: Update Traefik Values

File: repos/k8s-lab/traefik/values.yaml

Add file provider configuration via additionalConfigMaps:

additionalConfigMaps:
  hostname-auth:
    http:
      routers:
        # OAuth callback endpoint - must bypass auth (highest priority)
        auth-host-noauth:
          rule: "Host(`auth.lab.ctoaas.co`)"
          entryPoints:
            - websecure
          service: noop@internal
          priority: 300
          tls: {}
 
        # Internal hosts - no auth required (high priority)
        internal-host-noauth:
          rule: "HostRegexp(`{subdomain:.+}.lab.local.ctoaas.co`)"
          entryPoints:
            - websecure
          service: noop@internal
          priority: 200
          tls: {}
 
        # Public hosts with public=true label - require Google auth
        # Only matches ingresses that have opted-in via annotation:
        #   traefik.ingress.kubernetes.io/router.labels: public=true
        public-host-auth:
          rule: >
            HostRegexp(`{subdomain:.+}.lab.ctoaas.co`)
            && HeadersRegexp(`X-Forwarded-Labels`, `.*public=true.*`)
          entryPoints:
            - websecure
          middlewares:
            - google-auth
          service: noop@internal
          priority: 100
          tls: {}
 
      middlewares:
        google-auth:
          forwardAuth:
            address: "http://oauth2-proxy.oauth2-proxy.svc.cluster.local:4180/oauth2/auth"
            trustForwardHeader: true
            authResponseHeaders:
              - X-Auth-Request-User
              - X-Auth-Request-Email
              - X-Auth-Request-Access-Token
              - Authorization

Public Opt-In: Ingresses must add the annotation to be publicly accessible:

annotations:
  traefik.ingress.kubernetes.io/router.labels: public=true

Phase 4: Integration

Task 4.1: Add oauth2-proxy to Components

File: repos/k8s-lab/components/kustomization.yaml

Add to resources:

  - ../oauth2-proxy/

Task 4.2: Sync and Verify

# Apply changes (if using direct kustomize)
cd repos/k8s-lab
kustomize build components/ | kubectl apply -f -
kustomize build traefik/ | kubectl apply -f -
 
# Or if using ArgoCD, sync the applications
argocd app sync k8s-lab-components
argocd app sync k8s-lab-traefik

Phase 5: Validation

Task 5.1: Verify Secret Sync

# Check ClusterExternalSecret status
kubectl get clusterexternalsecrets google-oauth-credentials
 
# Verify secret created in oauth2-proxy namespace
kubectl get secret oauth2-proxy-secrets -n oauth2-proxy

Task 5.2: Verify oauth2-proxy

# Check pod status
kubectl get pods -n oauth2-proxy
 
# Check logs
kubectl logs -n oauth2-proxy -l app.kubernetes.io/name=oauth2-proxy
 
# Test internal connectivity
kubectl run -it --rm debug --image=curlimages/curl -- \
  curl -v http://oauth2-proxy.oauth2-proxy.svc.cluster.local:4180/ping

Task 5.3: Verify Traefik Configuration

# Check Traefik dashboard at https://traefik.lab.local.ctoaas.co
# Verify routers:
#   - auth-host-noauth@file (priority 300)
#   - internal-host-noauth@file (priority 200)
#   - public-host-auth@file (priority 100)
# Verify middleware:
#   - google-auth@file

Task 5.4: End-to-End Testing

TestCommand/ActionExpected Result
Public host with opt-inVisit https://service.lab.ctoaas.co (Ingress has public=true)Redirects to Google login
Public host without opt-inVisit https://service.lab.ctoaas.co (Ingress lacks annotation)No route match / not accessible
Auth callbackComplete Google loginRedirects back to original URL
Internal host bypassVisit https://traefik.lab.local.ctoaas.coDirect access, no auth
Auth host bypassVisit https://auth.lab.ctoaas.cooauth2-proxy responds (no redirect loop)
Email allowlistLogin with non-allowed emailAccess denied
Email allowlistLogin with allowed emailAccess granted

File Summary

New Files

FileDescription
repos/k8s-lab/oauth2-proxy/kustomization.yamlKustomize config with Helm chart
repos/k8s-lab/oauth2-proxy/oauth2-proxy-namespace.yamlNamespace with secret label
repos/k8s-lab/oauth2-proxy/values.yamloauth2-proxy Helm values
repos/k8s-lab/oauth2-proxy/allowed-emails.yamlConfigMap for email allowlist
repos/k8s-lab/components/central-secret-store/external-secrets/google-oauth.yamlClusterExternalSecret

Modified Files

FileChange
repos/k8s-lab/traefik/values.yamlAdd file provider with routers and middleware
repos/k8s-lab/components/central-secret-store/kustomization.yamlAdd google-oauth.yaml resource
repos/k8s-lab/components/kustomization.yamlAdd oauth2-proxy resource

Rollback Plan

If issues occur:

  1. Remove file provider config from traefik/values.yaml (restores pre-auth behavior)
  2. Remove oauth2-proxy from components/kustomization.yaml
  3. Delete oauth2-proxy namespace: kubectl delete namespace oauth2-proxy

Notes

  • The noop@internal service is a Traefik built-in that does nothing - it allows the request to continue to normal Ingress routing after middleware processing
  • Router priorities ensure more specific rules (auth host, internal hosts) are evaluated before the catch-all public host rule
  • The HostRegexp syntax changed in Traefik v3 - uses ^.+\\.domain$ format
  • Cookie domain .lab.ctoaas.co allows session sharing across all subdomains

Graph View