Techcle Wiki

Architecture

5 min read

System Architecture

Overview

The Market Making System follows a layered architecture with clear separation of concerns, designed for reliability, maintainability, and extensibility in a financial trading environment.

Architectural Principles

1. Interface Segregation

  • Abstract interfaces define contracts between components
  • Concrete implementations can be swapped without affecting other components
  • Enables testing with mocks and future multi-exchange support

2. Dependency Injection

  • Components receive dependencies through constructor injection
  • Enables loose coupling and testability
  • Configuration-driven component initialization

3. Asynchronous Operations

  • All external API calls are async to prevent blocking
  • Concurrent data collection from multiple sources
  • Non-blocking metrics aggregation

4. Configuration Separation

  • Environment-specific config (deployment, secrets) separate from market config
  • Market parameters stored in version-controlled repository
  • Runtime configuration validation

5. Error Resilience

  • Failures in one component don’t break the entire system
  • Graceful degradation with partial data collection
  • Comprehensive error logging and monitoring

Component Architecture

System Layers Overview

graph TB
    subgraph "API Layer"
        API[FastAPI Application]
    end
    
    subgraph "Business Logic Layer"
        MC[MetricsCollector]
        EI[ExchangeInterface]
        GMI[GridManagerInterface]
        REI[RegimeEngineInterface]
    end
    
    subgraph "Data Layer"
        MH[MetricsHistory]
        CL[ConfigurationLoader]
        CM[CredentialsManager]
    end
    
    subgraph "External Systems"
        KC[KuCoin API]
        N8N[n8n]
        ESO[External Secrets]
        K8S[Kubernetes]
        MM_DATA[market-maker-data]
    end
    
    API --> MC
    MC --> EI
    MC --> GMI
    MC --> REI
    MC --> MH
    MC --> CL
    MC --> CM
    
    EI --> KC
    CL --> MM_DATA
    CM --> ESO
    ESO --> K8S

API Layer Detail

graph TD
    A[HTTP Request] --> B[FastAPI Router]
    B --> C{Endpoint}
    C -->|"GET /health"| D[Health Check Handler]
    C -->|"POST /metrics/collect"| E[Metrics Collection Handler]
    
    D --> F[Return Health Status]
    E --> G[Validate Request]
    G --> H[Call MetricsCollector]
    H --> I[Format Response]
    I --> J[Return Collection Result]
    
    K[Pydantic Models] -.-> G
    L[Error Handling] -.-> F
    L -.-> I

Responsibilities:

  • HTTP request/response handling
  • Input validation using Pydantic models
  • Error response formatting
  • Request logging and correlation

Business Logic Layer Detail

graph TD
    A[MetricsCollector] --> B[ExchangeInterface]
    A --> C[GridManagerInterface]
    A --> D[RegimeEngineInterface]
    
    B --> E[KuCoinExchange<br/>Implementation]
    C --> F[GridManager<br/>Stub Implementation]
    D --> G[RegimeEngine<br/>Stub Implementation]
    
    E --> H[KuCoin API]
    F -.-> I[Future: Grid Logic]
    G -.-> J[Future: Technical Analysis]
    
    K[Data Aggregation] -.-> A
    L[Error Recovery] -.-> A
    M[Timing Logic] -.-> A

Key Components:

MetricsCollector

  • Pattern: Facade pattern for unified data access
  • Responsibilities:
    • Coordinate data collection from all sources
    • Implement timing logic (previous hour analysis)
    • Handle partial failures gracefully
    • Format data for storage and response

ExchangeInterface

  • Pattern: Abstract Factory + Strategy
  • Implementations: KuCoinExchange
  • Responsibilities:
    • Account balance queries
    • Market data retrieval
    • Historical data access
    • Grid information collection

GridManagerInterface

  • Current: Stub implementation
  • Future: Grid creation, monitoring, lifecycle management
  • Integration: Exchange behavior validation

RegimeEngineInterface

  • Current: Stub implementation
  • Future: Technical analysis, regime classification
  • Data Sources: Historical candles, volatility metrics

Data Layer Detail

graph TD
    A[MetricsHistory] --> B[YAML File Storage]
    A --> C[Git Integration]
    A --> D[Date-based Hierarchy]
    
    B --> E[Atomic Commits]
    C --> F[Version Control]
    D --> G["{year}/{month}/{day}/<br/>{timestamp}_{symbol}.yaml"]
    
    H[ConfigurationLoader] --> I["market-maker-data<br/>Repository"]
    H --> J[YAML Config Files]
    H --> K[Runtime Validation]
    
    L[CredentialsManager] --> M["External Secrets<br/>Operator"]
    L --> N[Kubernetes Secrets]
    L --> O[Runtime Injection]
    
    P[In-Memory Caching] -.-> H
    Q[No Persistence] -.-> L

Components:

MetricsHistory

  • Storage: YAML files with structured data
  • Organization: {year}/{month}/{day}/{timestamp}_{symbol}.yaml
  • Integration: Git commits for versioning
  • Querying: File-based access (no database)

ConfigurationLoader

  • Source: market-maker-data/ repository
  • Format: YAML configuration files
  • Validation: Runtime config validation
  • Caching: In-memory config caching

CredentialsManager

  • Source: External Secrets Operator
  • Storage: Kubernetes secrets
  • Access: Runtime injection
  • Security: No credential persistence in application

Data Flow

Metrics Collection Flow

sequenceDiagram
    participant N8N
    participant API
    participant Collector
    participant Exchange
    participant GridMgr
    participant RegimeEng
    participant History

    N8N->>API: POST /metrics/collect
    API->>Collector: collect_snapshot()
    Collector->>Exchange: get_account_balance()
    Collector->>Exchange: get_market_data()
    Collector->>GridMgr: get_grid_status()
    Collector->>RegimeEng: get_current_state()
    Collector->>History: store_snapshot()
    History-->>Collector: commit_result
    Collector-->>API: collection_result
    API-->>N8N: success_response

Initialization Flow

graph TD
    A[Application Startup] --> B[Load Environment Config]
    B --> C[Initialize Credentials Manager]
    C --> D[Initialize Configuration Loader]
    D --> E[Initialize Metrics History]
    E --> F[Initialize Exchange Interface]
    F --> G[Initialize Grid Manager]
    G --> H[Initialize Regime Engine]
    H --> I[Initialize Metrics Collector]
    I --> J[Ready for Requests]

Interface Definitions

ExchangeInterface

class ExchangeInterface(ABC):
    async def get_account_balance(self) -> AccountBalance
    async def get_market_data(self, symbol: str) -> MarketData
    async def get_historical_candles(self, symbol: str, interval: str, limit: int) -> List[HistoricalCandle]
    async def get_active_grids(self) -> List[GridInfo]

Data Models

@dataclass
class AccountBalance:
    total_usdt: float
    available_usdt: float
    locked_usdt: float
    deployable_capital: float
    reserve_required: float
 
@dataclass
class MarketData:
    symbol: str
    current_price: float
    change_24h_pct: float
    volume_24h: float
    last_updated: str
 
@dataclass
class RegimeState:
    verdict: str  # RANGE_OK, RANGE_WEAK, TRANSITION, TREND
    confidence_score: float
    analysis_timestamp: str
    metrics: Dict[str, Any]

Error Handling Architecture

Error Propagation

  • Component Level: Each component handles its own errors
  • Aggregation Level: MetricsCollector handles component failures
  • API Level: FastAPI handles request/response errors
  • Logging: Structured logging with correlation IDs

Failure Modes

  • Exchange Down: Continue with available data, log warning
  • Config Invalid: Fail fast on startup
  • Storage Failure: Return error status, don’t break collection
  • Partial Data: Collect what possible, mark incomplete

Security Architecture

Credential Management

  • Storage: External Secrets Operator in Kubernetes
  • Access: Runtime injection via environment variables
  • Scope: Limited to necessary API credentials
  • Rotation: Via ESO configuration updates

Data Protection

  • In Transit: HTTPS for all external communications
  • At Rest: Git encryption for sensitive config
  • Access Control: Kubernetes RBAC for infrastructure access

Deployment Architecture

Kubernetes Components

apiVersion: v1
kind: Service
metadata:
  name: metrics-service
spec:
  ports:
  - port: 80
    targetPort: 8000
  selector:
    app: metrics-service
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: metrics-service
spec:
  replicas: 1
  template:
    spec:
      containers:
      - name: metrics-service
        image: metrics-service:latest
        ports:
        - containerPort: 8000

Configuration Management

  • ConfigMaps: Application configuration
  • ExternalSecrets: API credentials
  • PersistentVolume: Not used (Git-based storage)
  • Service Mesh: Not implemented (single service)

Scalability Considerations

Current Limitations

  • Single Instance: No horizontal scaling
  • Memory Bound: All data loaded into memory
  • Sequential Processing: No parallel collection

Future Enhancements

  • Horizontal Scaling: Multiple collector instances
  • Database Storage: Replace file-based storage
  • Event-Driven: Replace polling with events
  • Caching Layer: Redis for frequently accessed data

Monitoring and Observability

Health Checks

  • Application: /health endpoint
  • Dependencies: Exchange API connectivity
  • Data: Configuration and credential access

Logging

  • Structured: JSON format with correlation IDs
  • Levels: INFO for operations, ERROR for failures
  • Retention: Container logs with rotation

Metrics

  • Collection Success: Success/failure rates
  • Data Quality: Completeness of collected data
  • Performance: Response times and throughput

Graph View