Techcle Wiki

Tasks

11 min read

Implementation Plan: Secure Proxy (sec-proxy)

Overview

This implementation plan breaks down the sec-proxy design into discrete coding tasks. The proxy will be built in Go, starting with core infrastructure, then adding registry handlers, rule engine, caching, and finally integration. Each task builds incrementally on previous work.

Tasks

  • 1. Project setup and core types

    • Initialize Go module with go mod init
    • Create directory structure: cmd/, internal/, pkg/, config/, tests/
    • Define core types: PackageIdentifier, PackageMetadata, VersionInfo, RuleResult
    • Set up logging with zap or zerolog
    • Requirements: 14.1

  • 2. Configuration system

    • 2.1 Implement YAML configuration parsing

      • Create ProxyConfig, GlobalConfig, RegistryConfig structs with yaml tags
      • Implement configuration file loading with gopkg.in/yaml.v3
      • Requirements: 6.1, 14.1

    • [ ]* 2.2 Write property test for configuration parsing

      • Property 6: Configuration Parsing
      • Validates: Requirements 6.1

    • 2.3 Implement configuration validation

      • Validate required fields (listenAddress, listenPort, registry URLs)
      • Validate rule references exist
      • Return specific error messages with field names
      • Requirements: 6.5, 14.2

    • [ ]* 2.4 Write property test for configuration validation

      • Property 10: Configuration Validation Errors
      • Validates: Requirements 6.5, 14.2

    • 2.5 Implement rule configuration parsing

      • Parse global rules and per-registry rules
      • Parse rule exclusions with package patterns
      • Support wildcard matching in package patterns
      • Requirements: 6.2, 6.3, 6.4

    • [ ]* 2.6 Write property tests for rule configuration

      • Property 7: Global Rule Application
      • Property 8: Registry Rule Override
      • Property 9: Rule Exclusions
      • Validates: Requirements 6.2, 6.3, 6.4

  • 3. Checkpoint - Configuration system complete

    • Ensure all tests pass, ask the user if questions arise.

  • 4. Rule engine foundation

    • 4.1 Implement Rule interface and evaluation context

      • Define Rule interface with ID(), Name(), Evaluate(), AppliesTo() methods
      • Implement EvaluationContext with current time and CVE database
      • Create RuleResult type
      • Requirements: 11.1

    • [ ]* 4.2 Write property test for rule interface

      • Property 24: Rule Interface Compatibility
      • Validates: Requirements 11.1

    • 4.3 Implement Age Rule

      • Create AgeRule struct implementing Rule interface
      • Calculate package age from publication timestamp
      • Compare against configured minimum age threshold
      • Support package exclusions with pattern matching
      • Requirements: 7.1, 7.2, 7.3, 7.4

    • [ ]* 4.4 Write property tests for Age Rule

      • Property 11: Publication Timestamp Retrieval
      • Property 12: Age-Based Filtering
      • Property 13: Age Rule Exclusions
      • Validates: Requirements 7.1, 7.2, 7.3, 7.4

    • [ ]* 4.5 Write unit tests for Age Rule edge cases

      • Test missing publication timestamp
      • Test zero age threshold
      • Test exclusion pattern matching
      • Requirements: 7.5

  • 5. Rule engine orchestration

    • 5.1 Implement rule evaluation orchestration

      • Apply rules in configured order
      • Combine multiple rule results with AND/OR logic
      • Handle rule evaluation errors gracefully
      • Continue evaluation if one rule fails
      • Requirements: 11.2, 11.3, 11.4

    • [ ]* 5.2 Write property tests for rule orchestration

      • Property 25: Rule Evaluation Order
      • Property 26: Rule Result Combination
      • Property 27: Rule Evaluation Error Handling
      • Validates: Requirements 11.2, 11.3, 11.4

  • 6. Checkpoint - Rule engine complete

    • Ensure all tests pass, ask the user if questions arise.

  • 7. Metadata cache implementation

    • 7.1 Implement in-memory metadata cache

      • Create MetadataCache interface
      • Implement in-memory cache with sync.Map
      • Support TTL-based expiration
      • Implement LRU eviction policy
      • Requirements: 12.1, 12.2

    • [ ]* 7.2 Write property tests for metadata caching

      • Property 3: Caching Metadata
      • Property 4: Metadata Cache Hit
      • Property 28: LRU Cache Eviction
      • Property 29: Cache Staleness
      • Validates: Requirements 1.3, 1.4, 12.1, 12.2

    • 7.3 Implement cache statistics and management

      • Track hit rate, entry count, memory usage
      • Implement manual cache invalidation
      • Requirements: 12.3, 12.4

    • [ ]* 7.4 Write property tests for cache management

      • Property 30: Manual Cache Invalidation
      • Property 31: Cache Statistics
      • Validates: Requirements 12.3, 12.4

    • 7.5 Implement Redis cache backend (optional)

      • Create Redis-backed implementation of MetadataCache
      • Support distributed caching across instances
      • Persist cache across restarts
      • Requirements: 12.5, 13.1

    • [ ]* 7.6 Write property tests for Redis cache

      • Property 32: Cache Persistence
      • Property 33: Distributed Cache Sharing
      • Validates: Requirements 12.5, 13.1

  • 8. Checkpoint - Caching complete

    • Ensure all tests pass, ask the user if questions arise.

  • 9. Docker registry handler

    • 9.1 Implement Docker registry handler

      • Create DockerHandler implementing RegistryHandler interface
      • Parse Docker registry URLs and image references
      • Fetch image manifests from Docker Registry HTTP API V2
      • Extract publication timestamps from image metadata
      • Requirements: 1.1, 9.2

    • [ ]* 9.2 Write property test for Docker handler

      • Property 1: Package Proxying
      • Property 19: Handler Routing
      • Validates: Requirements 1.1, 9.2

    • 9.3 Implement Docker version resolution

      • Resolve “latest” and other tags to specific digests
      • Fetch all available tags from registry
      • Apply age rules to select compliant version
      • Requirements: 8.1, 8.2, 8.3, 8.4, 8.5

    • [ ]* 9.4 Write property tests for Docker version resolution

      • Property 14: Version Metadata Retrieval
      • Property 15: Newest Compliant Version Selection
      • Property 16: Transparent Version Resolution
      • Property 17: No Compliant Version Blocking
      • Property 18: Specific Version Evaluation
      • Validates: Requirements 8.1, 8.2, 8.3, 8.4, 8.5

    • 9.5 Implement Docker image proxying

      • Stream image layers from upstream to client
      • Validate checksums during streaming
      • Handle authentication with upstream registry
      • Requirements: 1.2, 1.5

    • [ ]* 9.6 Write property tests for Docker validation

      • Property 2: Validation Before Serving
      • Property 5: Blocking Invalid Packages
      • Validates: Requirements 1.2, 1.5

  • 10. NPM registry handler

    • 10.1 Implement NPM registry handler

      • Create NPMHandler implementing RegistryHandler interface
      • Parse npm package URLs and scoped packages
      • Fetch package metadata from npm registry API
      • Extract publication timestamps from time field
      • Requirements: 2.1, 9.1

    • [ ]* 10.2 Write property test for NPM handler

      • Property 1: Package Proxying (npm variant)
      • Validates: Requirements 2.1, 9.1

    • 10.3 Implement NPM version resolution and validation

      • Resolve “latest” and other dist-tags
      • Apply age rules to select compliant version
      • Validate package integrity using checksums
      • Requirements: 2.2, 2.3, 2.4, 2.5

    • [ ]* 10.4 Write property tests for NPM validation

      • Property 2: Validation Before Serving (npm variant)
      • Property 5: Blocking Invalid Packages (npm variant)
      • Validates: Requirements 2.2, 2.5

  • 11. PyPI registry handler

    • 11.1 Implement PyPI registry handler

      • Create PipHandler implementing RegistryHandler interface
      • Parse pip package URLs
      • Fetch package metadata from PyPI JSON API
      • Extract publication timestamps from upload_time_iso_8601
      • Requirements: 3.1, 9.3

    • 11.2 Implement PyPI version resolution and validation

      • Resolve version specifiers to specific versions
      • Apply age rules to select compliant version
      • Verify cryptographic signatures if available
      • Requirements: 3.2, 3.3, 3.4, 3.5

    • [ ]* 11.3 Write property tests for PyPI handler

      • Property 1: Package Proxying (pip variant)
      • Property 2: Validation Before Serving (pip variant)
      • Validates: Requirements 3.1, 3.2, 3.5

  • 12. Maven registry handler

    • 12.1 Implement Maven registry handler

      • Create MavenHandler implementing RegistryHandler interface
      • Parse Maven artifact coordinates (groupId:artifactId:version)
      • Fetch maven-metadata.xml from repository
      • Extract version information and timestamps
      • Requirements: 4.1, 9.4

    • 12.2 Implement Maven version resolution and validation

      • Resolve LATEST and RELEASE to specific versions
      • Apply age rules to select compliant version
      • Validate POM files and checksums
      • Requirements: 4.2, 4.3, 4.4, 4.5

    • [ ]* 12.3 Write property tests for Maven handler

      • Property 1: Package Proxying (maven variant)
      • Property 2: Validation Before Serving (maven variant)
      • Validates: Requirements 4.1, 4.2, 4.5

  • 13. Go module proxy handler

    • 13.1 Implement Go module proxy handler

      • Create GoHandler implementing RegistryHandler interface
      • Implement Go module proxy protocol endpoints
      • Fetch module metadata from upstream proxy
      • Extract version timestamps from module info
      • Requirements: 5.1, 9.5

    • 13.2 Implement Go module version resolution and validation

      • Resolve version queries to specific versions
      • Apply age rules to select compliant version
      • Verify go.sum checksums
      • Requirements: 5.2, 5.3, 5.4, 5.5

    • [ ]* 13.3 Write property tests for Go handler

      • Property 1: Package Proxying (go variant)
      • Property 2: Validation Before Serving (go variant)
      • Validates: Requirements 5.1, 5.2, 5.5

  • 14. Checkpoint - All registry handlers complete

    • Ensure all tests pass, ask the user if questions arise.

  • 15. CVE database integration

    • 15.1 Implement CVE database client

      • Create CVEDatabase interface
      • Implement NVD API client for CVE queries
      • Parse CVE severity scores (CVSS)
      • Handle API rate limiting and errors
      • Requirements: 10.1

    • [ ]* 15.2 Write property test for CVE queries

      • Property 20: CVE Database Queries
      • Validates: Requirements 10.1

    • 15.3 Implement CVE Rule

      • Create CVERule struct implementing Rule interface
      • Query CVE database for package vulnerabilities
      • Evaluate severity against configured thresholds
      • Support blocking on critical/high severity CVEs
      • Requirements: 10.2, 10.3, 10.4

    • [ ]* 15.4 Write property tests for CVE Rule

      • Property 21: CVE Severity Evaluation
      • Property 22: Critical CVE Blocking
      • Property 23: Low Severity CVE Handling
      • Validates: Requirements 10.2, 10.3, 10.4

    • [ ]* 15.5 Write unit tests for CVE edge cases

      • Test CVE database unavailable
      • Test missing CVE data
      • Test malformed CVE responses
      • Requirements: 10.5

  • 16. Audit logging

    • 16.1 Implement audit logger

      • Create AuditLogger interface
      • Implement file-based audit logging
      • Log request events with package details
      • Log decision events with rule results
      • Log cache events (hit/miss/evict)
      • Requirements: 8.1, 8.2, 8.3, 8.4

    • [ ]* 16.2 Write property tests for audit logging

      • Property 41: Audit Logging for Requests
      • Property 42: Audit Logging for Decisions
      • Property 43: Audit Logging for Cache Events
      • Property 44: Audit Log Blocking Details
      • Validates: Requirements 8.1, 8.2, 8.3, 8.4

    • 16.3 Ensure audit log completeness

      • Include timestamps, client identity, package details
      • Include security outcomes and rule results
      • Support structured logging (JSON format)
      • Requirements: 8.5

    • [ ]* 16.4 Write property test for audit log completeness

      • Property 45: Audit Log Completeness
      • Validates: Requirements 8.5

  • 17. HTTP server and request routing

    • 17.1 Implement HTTP server

      • Create main HTTP server with net/http
      • Implement request routing to appropriate handlers
      • Support TLS configuration
      • Handle graceful shutdown
      • Requirements: 14.1, 14.5

    • [ ]* 17.2 Write property test for handler routing

      • Property 19: Handler Routing
      • Validates: Requirements 9.1, 9.2, 9.3, 9.4, 9.5

    • 17.3 Implement health check endpoint

      • Create /health endpoint
      • Report configuration status
      • Report cache status
      • Report handler status
      • Requirements: 14.5

    • [ ]* 17.4 Write property test for health check

      • Property 40: Health Check Endpoint
      • Validates: Requirements 14.5

    • 17.5 Implement configuration hot reload

      • Watch configuration file for changes
      • Reload configuration without dropping connections
      • Validate new configuration before applying
      • Requirements: 14.3

    • [ ]* 17.6 Write property test for hot reload

      • Property 38: Hot Configuration Reload
      • Validates: Requirements 14.3

  • 18. Error handling and fallback behavior

    • 18.1 Implement error types

      • Create UpstreamError, PolicyViolationError, ValidationError
      • Implement error response formatting
      • Include retryable flag in error responses
      • Requirements: 1.5, 2.5, 3.5, 4.5, 5.5

    • 18.2 Implement fallback behavior

      • Use cached metadata when upstream unavailable
      • Apply configured fallback for CVE database unavailable
      • Continue proxying without caching if cache backend fails
      • Requirements: 13.3

    • [ ]* 18.3 Write property test for offline operation

      • Property 35: Offline Metadata Operation
      • Validates: Requirements 13.3

    • 18.4 Implement circuit breaker for upstream registries

      • Track consecutive failures per registry
      • Open circuit after N failures
      • Attempt recovery after timeout
      • Requirements: 13.3

  • 19. Checkpoint - Core functionality complete

    • Ensure all tests pass, ask the user if questions arise.

  • 20. Performance optimization and metrics

    • 20.1 Implement Prometheus metrics

      • Export request rate, error rate, latency metrics
      • Export cache hit rate and size metrics
      • Export rule evaluation metrics
      • Create /metrics endpoint
      • Requirements: 13.2, 13.4

    • [ ]* 20.2 Write property tests for performance

      • Property 34: Metadata Lookup Response Time
      • Property 36: Concurrent Request Handling
      • Validates: Requirements 13.2, 13.4

    • 20.3 Implement connection pooling

      • Reuse HTTP connections to upstream registries
      • Configure connection pool sizes
      • Handle connection timeouts
      • Requirements: 13.4

    • 20.4 Implement streaming for large packages

      • Stream package data without buffering in memory
      • Support range requests
      • Handle partial content responses
      • Requirements: 13.4

  • 21. Integration and end-to-end testing

    • [ ]* 21.1 Write integration tests

      • Test complete request flow: client → proxy → upstream
      • Test age rule blocking with real metadata
      • Test version resolution with multiple versions
      • Test cache hit/miss scenarios
      • Requirements: All

    • [ ]* 21.2 Write multi-registry integration tests

      • Test Docker, npm, pip, Maven, Go handlers together
      • Test global vs per-registry rules
      • Test rule exclusions across registries
      • Requirements: 6.2, 6.3, 6.4

  • 22. Documentation and deployment

    • 22.1 Create example configuration files

      • Provide example YAML with common use cases
      • Document all configuration options
      • Include comments explaining each setting
      • Requirements: 14.4

    • [ ]* 22.2 Write property test for configuration completeness

      • Property 39: Configuration Completeness
      • Validates: Requirements 14.4

    • 22.3 Create Dockerfile

      • Build multi-stage Docker image
      • Use minimal base image (alpine or distroless)
      • Include health check
      • Requirements: 13.5

    • 22.4 Create Kubernetes manifests

      • Create Deployment with multiple replicas
      • Create Service for load balancing
      • Create ConfigMap for configuration
      • Include liveness and readiness probes
      • Requirements: 13.5

  • 23. Final checkpoint - Complete implementation

    • Ensure all tests pass, ask the user if questions arise.

Notes

  • Tasks marked with * are optional and can be skipped for faster MVP
  • Each task references specific requirements for traceability
  • Checkpoints ensure incremental validation
  • Property tests validate universal correctness properties
  • Unit tests validate specific examples and edge cases
  • Integration tests validate end-to-end flows

Graph View