Data Schema Specification

Project: Grid Exit Strategy - Regime Management System
Version: 1.0
Last Updated: 2026-02-01

Overview

All data persisted as structured files in Git repository (market-maker-data/). This document defines the schema for each file type to ensure validation and consistency.

File Structure

market-maker-data/
├── metrics/
│   └── {symbol}/
│       └── {YYYY-MM-DD}-{HH}.yaml          # Hourly metrics
├── exit_states/
│   └── {symbol}/
│       └── {YYYY-MM-DD}.json               # Daily state transitions
├── decisions/
│   └── {YYYY-MM-DD}/
│       └── dec-{symbol}-{HHMMSS}.yaml      # Decision records
└── dashboards/                              # Static HTML/JS/CSS (TBD)

1. Metrics Files Schema

Location: metrics/{symbol}/{YYYY-MM-DD}-{HH}.yaml

Purpose: Hourly regime evaluation results with all calculated metrics

Schema Definition

symbol: string                    # e.g., "ETH-USDT"
timestamp: string                 # ISO 8601 UTC, e.g., "2026-02-01T14:00:00Z"
regime: enum                      # RANGE_OK | RANGE_WEAK | TRANSITION | TREND
confidence: float                 # 0.0 to 1.0
exit_state: enum                  # NORMAL | WARNING | LATEST_ACCEPTABLE_EXIT | MANDATORY_EXIT
 
detailed_analysis:
  adx:
    current: float                # 0 to 100
    trend: enum                   # RISING | FALLING | STABLE
    slope: float                  # Rate of change
  
  efficiency_ratio:
    current: float                # 0 to 1
  
  autocorrelation:
    lag1: float                   # -1 to 1
  
  ou_process:
    halflife_hours: float         # Positive value, or null if non-stationary
  
  slope:
    normalized: float             # Price slope / ATR
  
  bollinger:
    bandwidth: float              # (upper - lower) / middle
    trend: enum                   # EXPANDING | CONTRACTING | STABLE
    slope: float                  # Rate of change
 
gate_evaluation:                  # Only present if grid is stopped
  gate1_directional_energy:
    status: enum                  # PASSED | FAILED | NOT_EVALUATED
    details: string               # Explanation
  
  gate2_mean_reversion:
    status: enum                  # PASSED | FAILED | NOT_EVALUATED
    details: string
  
  gate3_tradable_volatility:
    status: enum                  # PASSED | FAILED | NOT_EVALUATED
    details: string
 
grid_id: string                   # e.g., "eth-grid-1"
config_version: string            # Git commit hash of config used

Validation Rules

Field	Rule
`symbol`	Required, non-empty string
`timestamp`	Required, valid ISO 8601 UTC
`regime`	Required, must be one of: RANGE_OK, RANGE_WEAK, TRANSITION, TREND
`confidence`	Required, 0.0 ≤ value ≤ 1.0
`exit_state`	Required, must be valid exit state enum
`adx.current`	Required, 0 ≤ value ≤ 100
`efficiency_ratio.current`	Required, 0 ≤ value ≤ 1
`autocorrelation.lag1`	Required, -1 ≤ value ≤ 1
`ou_process.halflife_hours`	Optional (null if non-stationary), if present must be > 0
`gate_evaluation`	Optional, only present if grid stopped

Example

symbol: ETH-USDT
timestamp: "2026-02-01T14:00:00Z"
regime: TRANSITION
confidence: 0.54
exit_state: WARNING
 
detailed_analysis:
  adx:
    current: 32.5
    trend: RISING
    slope: 0.8
  
  efficiency_ratio:
    current: 0.68
  
  autocorrelation:
    lag1: -0.12
  
  ou_process:
    halflife_hours: 28.4
  
  slope:
    normalized: 0.15
  
  bollinger:
    bandwidth: 0.041
    trend: EXPANDING
    slope: 0.007
 
grid_id: eth-grid-1
config_version: a3f8d92e

2. Exit State Transition Files Schema

Location: exit_states/{symbol}/{YYYY-MM-DD}.json

Purpose: Daily log of exit state transitions with timestamps and reasoning

Schema Definition

{
  "symbol": "string",
  "grid_id": "string",
  "date": "YYYY-MM-DD",
  "transitions": [
    {
      "timestamp": "ISO 8601 UTC",
      "from_state": "enum: NORMAL | WARNING | LATEST_ACCEPTABLE_EXIT | MANDATORY_EXIT",
      "to_state": "enum: NORMAL | WARNING | LATEST_ACCEPTABLE_EXIT | MANDATORY_EXIT",
      "reasons": ["string", "string"],
      "regime_verdict": "enum: RANGE_OK | RANGE_WEAK | TRANSITION | TREND",
      "confidence": "float 0.0-1.0",
      "metrics": {
        "adx": "float",
        "efficiency_ratio": "float",
        "autocorrelation": "float",
        "ou_halflife": "float or null",
        "slope": "float",
        "bollinger_bandwidth": "float"
      }
    }
  ],
  "last_notification": {
    "WARNING": "ISO 8601 UTC or null",
    "LATEST_ACCEPTABLE_EXIT": "ISO 8601 UTC or null",
    "MANDATORY_EXIT": "ISO 8601 UTC or null"
  }
}

Validation Rules

Field	Rule
`symbol`	Required, non-empty string
`grid_id`	Required, non-empty string
`date`	Required, YYYY-MM-DD format
`transitions`	Required array, may be empty
`transitions[].timestamp`	Required, valid ISO 8601 UTC
`transitions[].from_state`	Required, valid exit state enum
`transitions[].to_state`	Required, valid exit state enum
`transitions[].reasons`	Required, non-empty array of strings
`transitions[].confidence`	Required, 0.0 ≤ value ≤ 1.0
`last_notification.*`	Valid ISO 8601 UTC or null

Additional Rules:

Transitions must be chronologically ordered
Each transition must have at least one reason
Last notification timestamps used for rate limiting

Example

{
  "symbol": "ETH-USDT",
  "grid_id": "eth-grid-1",
  "date": "2026-02-01",
  "transitions": [
    {
      "timestamp": "2026-02-01T09:15:00Z",
      "from_state": "NORMAL",
      "to_state": "WARNING",
      "reasons": [
        "TRANSITION probability rising (0.45)",
        "Regime confidence declining over 3 bars"
      ],
      "regime_verdict": "RANGE_WEAK",
      "confidence": 0.48,
      "metrics": {
        "adx": 28.5,
        "efficiency_ratio": 0.62,
        "autocorrelation": -0.08,
        "ou_halflife": 32.1,
        "slope": 0.12,
        "bollinger_bandwidth": 0.038
      }
    },
    {
      "timestamp": "2026-02-01T12:00:00Z",
      "from_state": "WARNING",
      "to_state": "LATEST_ACCEPTABLE_EXIT",
      "reasons": [
        "TRANSITION persistence confirmed (4h bars: 2 consecutive)"
      ],
      "regime_verdict": "TRANSITION",
      "confidence": 0.42,
      "metrics": {
        "adx": 35.2,
        "efficiency_ratio": 0.72,
        "autocorrelation": 0.05,
        "ou_halflife": null,
        "slope": 0.18,
        "bollinger_bandwidth": 0.045
      }
    }
  ],
  "last_notification": {
    "WARNING": "2026-02-01T09:15:00Z",
    "LATEST_ACCEPTABLE_EXIT": "2026-02-01T12:00:00Z",
    "MANDATORY_EXIT": null
  }
}

3. Decision Record Files Schema

Location: decisions/{YYYY-MM-DD}/dec-{symbol}-{HHMMSS}.yaml

Purpose: Immutable decision records for audit trail

Schema Definition

decision_id: string               # Unique identifier
symbol: string
grid_id: string
timestamp: string                 # ISO 8601 UTC when decision created
 
recommendation:                   # IMMUTABLE after creation
  action: enum                    # RUN_GRID | RUN_GRID_NO_SCALE | STOP_GRID | CREATE_GRID | IGNORE | SNOOZE
  regime_verdict: enum            # RANGE_OK | RANGE_WEAK | TRANSITION | TREND
  exit_state: enum                # NORMAL | WARNING | LATEST_ACCEPTABLE_EXIT | MANDATORY_EXIT
  confidence: float               # 0.0 to 1.0
  reasoning: string               # Human-readable explanation
  config_version: string          # Git commit hash
  validity_window_end: string     # ISO 8601 UTC
 
action_records:                   # Appended over time
  - timestamp: string             # ISO 8601 UTC
    action_taken: enum            # USER_ACCEPTED | USER_DECLINED | SYSTEM_TIMEOUT | SYSTEM_AUTOMATED
    user_notes: string            # Optional
 
evaluation_records:               # Appended at 24h, 72h, 7d intervals
  - evaluation_timestamp: string  # ISO 8601 UTC
    horizon: string               # "24h" | "72h" | "7d"
    regime_correctness: boolean   # Was regime classification correct?
    economic_impact_usd: float    # Profit/loss from this decision
    post_exit_adverse_move: float # If exited, how much further did price move against position?
    notes: string

Validation Rules

Field	Rule
`decision_id`	Required, unique identifier
`symbol`	Required, non-empty string
`timestamp`	Required, valid ISO 8601 UTC
`recommendation`	Required, immutable after creation
`recommendation.action`	Required, valid action enum
`recommendation.confidence`	Required, 0.0 ≤ value ≤ 1.0
`action_records`	Optional array, chronologically ordered
`evaluation_records`	Optional array, chronologically ordered

Additional Rules:

Recommendation section is immutable after file creation
Action and evaluation records can only be appended
Timestamps must be chronologically ordered
Configuration version hash must reference valid config

Example

decision_id: dec-2026-02-01-091500-eth
symbol: ETH-USDT
grid_id: eth-grid-1
timestamp: "2026-02-01T09:15:00Z"
 
recommendation:
  action: STOP_GRID
  regime_verdict: TRANSITION
  exit_state: WARNING
  confidence: 0.48
  reasoning: "TRANSITION probability rising (0.45) + regime confidence declining over 3 bars. Early exit recommended to preserve capital."
  config_version: a3f8d92e
  validity_window_end: "2026-02-01T21:15:00Z"
 
action_records:
  - timestamp: "2026-02-01T10:45:00Z"
    action_taken: USER_ACCEPTED
    user_notes: "Agreed - trend forming, better to exit now than wait"
 
evaluation_records:
  - evaluation_timestamp: "2026-02-02T09:15:00Z"
    horizon: "24h"
    regime_correctness: true
    economic_impact_usd: 47.50
    post_exit_adverse_move: 127.80
    notes: "Regime remained TRANSITION, then TREND. Exit preserved $127.80 of potential loss."

4. Configuration Schema

Location: config/exit_strategy_config.yaml

Purpose: All configurable thresholds and parameters

Schema Definition

exit_rules:
  latest_acceptable_exit:
    transition_persistence_4h_bars: integer        # Default: 2
    transition_persistence_1h_bars: integer        # Default: 4
    mean_reversion_halflife_multiplier: float      # Default: 2.0
    volatility_expansion_threshold: float          # Default: 1.25
    zscore_reversion_failure_bars: integer         # Default: 6
  
  warning:
    minimum_conditions_required: integer           # MUST BE >= 2, Default: 2
    transition_probability_threshold: float        # Default: 0.40
    regime_confidence_decline_bars: integer        # Default: 3
    efficiency_ratio_threshold: float              # Default: 0.6
    mean_reversion_slowdown_threshold: float       # Default: 1.5
    volatility_expansion_min: float                # Default: 1.10
    volatility_expansion_max: float                # Default: 1.25
  
  mandatory_exit:
    consecutive_closes_outside_range: integer      # Default: 2
    directional_swing_bars: integer                # Default: 6
    stop_loss_buffer_atr: float                    # Default: 0.1
 
notifications:
  rate_limits:
    warning_min_hours: integer                     # Default: 4
    latest_acceptable_min_hours: integer           # Default: 2
    mandatory_min_hours: integer                   # Default: 1
 
gates:
  gate1_directional_energy:
    trend_score_threshold: float                   # Max TrendScore to pass
    adx_threshold: float                           # Max ADX to pass
    passage_logic: string                          # "consecutive" | "majority"
  
  gate2_mean_reversion:
    mean_rev_score_threshold: float                # Min MeanRevScore to pass
    autocorrelation_threshold: float               # Max autocorrelation (should be <= 0)
    ou_halflife_max_hours: float                   # Max half-life to pass
  
  gate3_tradable_volatility:
    atr_percent_min: float                         # Min volatility
    atr_percent_max: float                         # Max volatility
    bollinger_bandwidth_stable: boolean            # Require stable/shrinking BB?

Validation Rules

Field	Rule
`warning.minimum_conditions_required`	MUST be >= 2 (critical design requirement)
All `*_threshold` floats	Must be within valid metric ranges
All `*_hours` integers	Must be positive
All `*_bars` integers	Must be positive
`gates.*.passage_logic`	Must be “consecutive” or “majority”

Additional Rules:

Configuration changes tracked via Git commits
Configuration version hash included in all decision records
Schema validation on load (fail fast if invalid)

5. Implementation Requirements

Pydantic Models

Create Python models for all schemas:

# src/schemas/metrics.py
from pydantic import BaseModel, Field
from enum import Enum
from datetime import datetime
 
class RegimeType(str, Enum):
    RANGE_OK = "RANGE_OK"
    RANGE_WEAK = "RANGE_WEAK"
    TRANSITION = "TRANSITION"
    TREND = "TREND"
 
class ExitState(str, Enum):
    NORMAL = "NORMAL"
    WARNING = "WARNING"
    LATEST_ACCEPTABLE_EXIT = "LATEST_ACCEPTABLE_EXIT"
    MANDATORY_EXIT = "MANDATORY_EXIT"
 
class ADXMetric(BaseModel):
    current: float = Field(ge=0, le=100)
    trend: str = Field(pattern="^(RISING|FALLING|STABLE)$")
    slope: float
 
class MetricsFile(BaseModel):
    symbol: str
    timestamp: datetime
    regime: RegimeType
    confidence: float = Field(ge=0.0, le=1.0)
    exit_state: ExitState
    detailed_analysis: dict
    grid_id: str
    config_version: str
    
    class Config:
        json_encoders = {
            datetime: lambda v: v.isoformat()
        }

Validation Strategy

Pre-Commit Validation:

def validate_and_commit(file_path: Path, data: BaseModel):
    """Validate schema before Git commit"""
    try:
        # Pydantic validates on model creation
        yaml_content = data.model_dump_json()
        
        # Write to file
        file_path.write_text(yaml_content)
        
        # Git commit
        repo.git.add(str(file_path))
        repo.git.commit(m=f"[{data.symbol}] Update {file_path.stem}")
        
    except ValidationError as e:
        logger.error(f"Schema validation failed: {e}")
        raise

Runtime Validation:

def load_metrics_file(file_path: Path) -> MetricsFile:
    """Load and validate metrics file"""
    raw_data = yaml.safe_load(file_path.read_text())
    return MetricsFile(**raw_data)  # Pydantic validates

6. Schema Evolution

Versioning Strategy

Schema Version Tracking:

Schema version included in file metadata (optional field)
Backward compatibility maintained for reading old files
Migration scripts provided for major schema changes

Example Migration:

def migrate_v1_to_v2(old_metrics: dict) -> MetricsFile:
    """Migrate old schema to new schema"""
    if old_metrics.get("schema_version") == "1.0":
        # Add new required fields with defaults
        old_metrics["exit_state"] = "NORMAL"
        old_metrics["schema_version"] = "2.0"
    
    return MetricsFile(**old_metrics)

Change Log

2026-02-01: Initial schema definition (v1.0)

Techcle Wiki

Explorer

SCHEMA

Data Schema Specification

Overview

File Structure

1. Metrics Files Schema

Schema Definition

Validation Rules

Example

2. Exit State Transition Files Schema

Schema Definition

Validation Rules

Example

3. Decision Record Files Schema

Schema Definition

Validation Rules

Example

4. Configuration Schema

Schema Definition

Validation Rules

5. Implementation Requirements

Pydantic Models

Validation Strategy

6. Schema Evolution

Versioning Strategy

Change Log

Graph View

Table of Contents