Techcle Wiki

Secret Management

4 min read

Secret Management Best Practices

Rule: Never Create Secrets Manually

NEVER create or modify secrets manually. Always use automated systems (ESO) to sync from central secret locations.

Central Secret Management Architecture

Our cluster uses External Secrets Operator (ESO) with a centralized secret management approach:

  • Central Store: central-secret-store namespace contains master secrets
  • Distribution: ClusterExternalSecrets automatically sync to labeled namespaces
  • Store Provider: central-secret-store ClusterSecretStore using Kubernetes provider
  • Consistency: All environments use identical credential sources

Available Cluster Secrets

Core Infrastructure Secrets

1. GitHub Container Registry (application-docker-registry)

  • Source Secret: github-pat in central-secret-store
  • Target Secret: gh-docker-registry-creds (type: kubernetes.io/dockerconfigjson)
  • Namespace Label: secrets/gh-docker-registry=true
  • Usage: Pull private images from ghcr.io
  • Distributed to: backstage, backstage-catalog-api, code-server, image-factory-kargo, ai-dev, n8n, n8n-operator-system

2. GitHub OAuth Credentials (github-oauth-credentials)

  • Source Secret: github-oauth in central-secret-store
  • Target Secret: gh-oauth (type: Opaque)
  • Namespace Label: secrets/gh-oauth-credentials=true
  • Keys: GITHUB_CLIENT_ID, GITHUB_CLIENT_SECRET, GITHUB_TOKEN
  • Usage: OAuth authentication with GitHub
  • Distributed to: backstage, market-making

3. GitHub Git Credentials (kargo-git-credentials)

  • Source Secret: github-pat in central-secret-store
  • Target Secret: git-credentials (type: Opaque)
  • Namespace Label: secrets/gh-git-credentials=true
  • Usage: Git operations in Kargo pipelines
  • Distributed to: argocd, backstage-kargo, image-factory-kargo

Platform Secrets

4. Kargo Admin Credentials (kargo-admin-credentials)

  • Source Secret: kargo-admin-credentials in central-secret-store
  • Target Secret: kargo-admin-credentials (type: Opaque)
  • Namespace Label: secrets/kargo-admin-credentials=true
  • Usage: Kargo administrative access
  • Distributed to: kargo, kargo-cluster-secrets

5. Cloudflare API Token (cloudflare-api-token)

  • Source Secret: cloudflare-api-token in central-secret-store
  • Target Secret: cloudflare-api-token (type: Opaque)
  • Namespace Label: secrets/cloudflare-api-token=true
  • Usage: DNS management and SSL certificate provisioning
  • Distributed to: cert-manager

6. Code Server SSH Key (code-server-ssh-key)

  • Source Secret: code-server-ssh-key in central-secret-store
  • Target Secret: code-server-ssh-key (type: Opaque)
  • Usage: SSH access for code-server workspace
  • Distributed to: code-server

Application-Specific Secrets

7. Market Making KuCoin Credentials (metrics-service-credentials)

  • Source Secret: market-making-kucoin-credentials in central-secret-store
  • Target Secret: metrics-service-credentials (type: Opaque)
  • Usage: KuCoin API access for market making service
  • Distributed to: market-making

How to Use Cluster Secrets

For New Applications

  1. Label your namespace with the appropriate secret label:

    apiVersion: v1
kind: Namespace
metadata:
  name: my-app
  labels:
    secrets/gh-docker-registry: "true"  # For GHCR access
    secrets/gh-oauth-credentials: "true"  # For GitHub OAuth

  2. Reference the secret in your deployment:

    spec:
  imagePullSecrets:
  - name: gh-docker-registry-creds  # Automatically synced
  containers:
  - name: app
    env:
    - name: GITHUB_TOKEN
      valueFrom:
        secretKeyRef:
          name: gh-oauth
          key: GITHUB_TOKEN

Available Secret Labels

LabelSecret CreatedTypeUsage
secrets/gh-docker-registry=truegh-docker-registry-credsdockerconfigjsonPull from ghcr.io
secrets/gh-oauth-credentials=truegh-oauthOpaqueGitHub OAuth
secrets/gh-git-credentials=truegit-credentialsOpaqueGit operations
secrets/kargo-admin-credentials=truekargo-admin-credentialsOpaqueKargo admin
secrets/cloudflare-api-token=truecloudflare-api-tokenOpaqueDNS/SSL
secrets/talos-config=truetalos-configOpaquetalosctl cluster ops (etcd defrag)

Secret Refresh and Monitoring

  • Refresh Interval: Most secrets refresh every 1 hour
  • Status Monitoring: Check ExternalSecret status with kubectl get externalsecrets -A
  • Troubleshooting: View ESO controller logs with task platform:logs

Secret Management Workflow

Adding New Central Secrets

  1. Create secret in central-secret-store namespace:

    kubectl create secret generic my-new-secret \
  --from-literal=key1=value1 \
  --from-literal=key2=value2 \
  -n central-secret-store

  2. Create ClusterExternalSecret to distribute:

    apiVersion: external-secrets.io/v1beta1
kind: ClusterExternalSecret
metadata:
  name: my-new-secret
spec:
  namespaceSelector:
    matchLabels:
      secrets/my-new-secret: "true"
  externalSecretSpec:
    secretStoreRef:
      kind: ClusterSecretStore
      name: central-secret-store
    target:
      name: my-new-secret
    data:
    - secretKey: key1
      remoteRef:
        key: my-new-secret
        property: key1

  3. Label target namespaces:

    kubectl label namespace my-app secrets/my-new-secret=true

Why This Architecture Matters

  1. Security: Single source of truth for credentials
  2. Consistency: All environments use identical credentials
  3. Maintainability: Update once, propagate everywhere
  4. Automation: No manual intervention required
  5. Auditability: Clear credential lifecycle and usage
  6. Scalability: Easy to add new secrets and target namespaces

What NOT to Do

  • ❌ Create secrets manually with kubectl create secret
  • ❌ Copy/paste credentials between namespaces
  • ❌ Hardcode credentials in manifests or scripts
  • ❌ Hardcode credentials in ESO policies
  • ❌ Create environment-specific credential variations
  • ❌ Modify secrets directly in target namespaces

What TO Do

  • ✅ Use namespace labels to request secret distribution
  • ✅ Create secrets in central-secret-store namespace
  • ✅ Use ClusterExternalSecrets for distribution
  • ✅ Monitor secret sync status regularly
  • ✅ Follow the established naming conventions
  • ✅ Use consistent labeling for secret identification

Troubleshooting

Common Issues

  1. Secret not appearing in namespace:

    • Check namespace has correct label
    • Verify ClusterExternalSecret exists
    • Check ExternalSecret status: kubectl get externalsecret -n <namespace>

  2. Secret sync errors:

    • Check ESO controller logs: task platform:logs
    • Verify source secret exists in central-secret-store
    • Check ClusterSecretStore status

  3. Permission issues:

    • Verify ESO service account has proper RBAC
    • Check ClusterSecretStore configuration

Useful Commands

# Check all external secrets status
kubectl get externalsecrets -A
 
# Check cluster secret stores
kubectl get clustersecretstores
 
# Check what secrets are distributed where
kubectl get secrets -A -l managed-by=external-secrets
 
# View ESO controller logs
task platform:logs
 
# Check platform status
task platform:status

Graph View