Market-Maker-MVP Codebase Deep Dive: Data Flow & Issues Map

Investigation Date: 2026-02-12
Scope: Comprehensive trace of data flows, hardcoded values, missing calculations, and systemic issues

Executive Summary

This investigation reveals 5 major systemic issues across the market-maker-mvp codebase:

Integer Conversion Bug - Values multiplied by 100/10000000 are WRITTEN to YAML but NEVER divided back when read
Fake Historical Data - Repeated scalar values ([adx] * 10) bypass real calculations
Hardcoded Fallback Values - .get("key", DEFAULT) patterns hide missing data
Missing Baseline Calculations - TODOs for 7-day rolling averages, but no implementation
Static Notification System - Notifications use hardcoded logic, ignore calculated metrics

1. Integer Conversion Data Flow

1.1 Where Values Are Written (Integer Conversion)

Location: src/grid/configuration_manager.py:270-308

# Lines 270-282: Grid configuration snapshot
"price_range": {
    "upper_bound": int(config.price_range.upper_bound * 100),  # ❌ BUG
    "lower_bound": int(config.price_range.lower_bound * 100),  # ❌ BUG
    "entry_price": int(config.price_range.entry_price * 100),  # ❌ BUG
},
"grid_structure": {
    "amount_per_grid": int(config.grid_structure.amount_per_grid * 10000000),  # ❌ BUG
},
"profit_configuration": {
    "profit_per_grid_min": int(config.profit_configuration.profit_per_grid_min * 100),  # ❌ BUG
    "profit_per_grid_max": int(config.profit_configuration.profit_per_grid_max * 100),  # ❌ BUG
    "profit_per_grid_avg": int(config.profit_configuration.profit_per_grid_avg * 100),  # ❌ BUG
}

Also in:

src/metrics/collector.py:644-649 - Minute prices stored as int(price * 100)
src/metrics/collector.py:884-887 - OHLC candles stored as int(price * 100)
src/metrics/collector.py:1193 - Stop loss distance stored as int((current_price - stop_loss_price) * 100)
src/regime/engine.py:1969-1970 - ATR stored as int(volatility * estimated_price * 100)
src/regime/engine.py:2709-2736 - Discovered range and current price stored as integers
src/regime/grid_comparison.py:200-230 - Grid comparison values stored as integers
src/regime/range_discovery.py:563-577 - Range discovery results stored as integers

Impact: ~15 locations write integer-converted values to YAML/JSON

1.2 Where Values Should Be Read Back (BUT AREN’T)

Critical Finding: There is NO corresponding division in the read path!

Expected pattern (MISSING):

# ❌ DOES NOT EXIST
upper_bound = config_snapshot["price_range"]["upper_bound"] / 100
lower_bound = config_snapshot["price_range"]["lower_bound"] / 100

Actual read patterns:

# In send_regime_notifications.py and evaluators - they read YAML directly
regime_analysis = analysis.get("regime_analysis", {})
confidence = regime_analysis.get("confidence", 0.0)  # ❌ No conversion

Result: Integer values are stored but read as-is, resulting in:

Prices displayed as 320000 instead of 3200.00
Amounts displayed as 15000000 instead of 0.0015
Metrics calculations using wrong scale

1.3 Partial Fix in history.py

Location: src/metrics/history.py:199-207

def _price_int_to_float(self, maybe_price: Any) -> Optional[float]:
    """Convert stored integer (price*100) or float to float price."""
    if isinstance(maybe_price, int) and maybe_price > 10000:  # ⚠️ Heuristic!
        return maybe_price / 100.0
    return float(maybe_price)

Problems:

Only used in _compute_grid_ladder() - not used globally
Uses heuristic (> 10000) which could fail for low prices
No handling for amount_per_grid (10000000 multiplier)
Comment says “market summary stores prices as integer cents” but no systematic conversion

1.4 Impact on Notifications

Location: send_regime_notifications.py:91-230

The notification system reads from YAML but has NO awareness of integer storage:

# Lines 103-128: Extracts metrics without conversion
range_analysis = regime_metrics.get("range_analysis", {})
vol_metrics = regime_metrics.get("volatility_metrics", {})
mean_rev = regime_metrics.get("mean_reversion", {})
 
# Uses values directly - if they're integers, they're WRONG
adx = detailed.get("adx", {}).get("current", 25.0)
vol_expansion = vol_metrics.get("volatility_expansion_ratio", 1.0)

Result: Notifications display wrong values or use wrong values in calculations

2. Fake Historical Data Usage

2.1 Where Fake Data Is Created

Location: src/regime/engine.py:286-299, 373-385

# Line 286: Creates fake ADX history
adx_history = [adx] * 10  # TODO Phase 2: Load from previous YAML files
 
# Line 295: Creates fake ATR history  
atr = 1500.0  # Will be replaced when we store ATR in detailed_analysis
atr_history = [atr] * 100  # TODO Phase 2: Load from previous YAML files
 
# Line 299: Creates fake Bollinger bandwidth history
bb_bandwidth_history = [bb_bandwidth] * 10  # TODO Phase 2: Load from previous YAML files

Pattern: Takes a single current value and repeats it N times to create “history”

2.2 Where Fake Data Is Used

Restart Gates Evaluation: src/regime/engine.py:307-324, 393-410

# These functions receive fake history arrays
verdict, gate_results = evaluate_restart_gates(
    price_data=price_data,
    trend_score=trend_score,
    mean_rev_score=mean_rev_score,
    adx=adx,
    adx_history=adx_history,  # ❌ Fake: [25.0, 25.0, 25.0, ...]
    normalized_slope=normalized_slope,
    efficiency_ratio=efficiency_ratio,
    lag1_autocorr=lag1_autocorr,
    ou_half_life=ou_half_life,
    atr=atr,
    atr_history=atr_history,  # ❌ Fake: [1500, 1500, 1500, ...]
    bb_bandwidth=bb_bandwidth,
    bb_bandwidth_history=bb_bandwidth_history,  # ❌ Fake: [0.02, 0.02, 0.02, ...]
    gate_1_config=gate_1_config,
    gate_2_config=gate_2_config,
    gate_3_config=gate_3_config,
)

2.3 Impact on Calculations

Gate 1: Directional Energy Decay

Checks if ADX is declining over time
With fake history [25, 25, 25], ADX NEVER declines
Gate 1 can never pass if ADX is high

Gate 3: Tradable Volatility

Checks if volatility has stabilized
With fake ATR history [1500, 1500, 1500], volatility NEVER changes
Gate 3 always sees “stable” volatility regardless of reality

Confidence Calculation:

Fake history makes all historical comparisons meaningless
Trends cannot be detected
Expansions cannot be measured

2.4 What Real History Would Look Like

Expected (from MetricsHistoryLoader): src/exit_strategy/history_loader.py:70-100

# Should load actual historical YAML files:
# metrics/2025/01/15/14_ETH-USDT.yaml
# metrics/2025/01/15/13_ETH-USDT.yaml
# metrics/2025/01/15/12_ETH-USDT.yaml
# ...
 
# Extract ADX from each file:
adx_history = [
    metrics_file_14["analysis"]["regime_analysis"]["adx"]["current"],  # 28.5
    metrics_file_13["analysis"]["regime_analysis"]["adx"]["current"],  # 27.2
    metrics_file_12["analysis"]["regime_analysis"]["adx"]["current"],  # 25.8
    ...
]

But this is NOT implemented - TODO comments everywhere

3. Hardcoded Fallback Values

3.1 Pattern: `.get("key", HARDCODED_DEFAULT)`

Locations with hardcoded defaults:

In send_regime_notifications.py:

# Line 117
range_conf = range_analysis.get("range_confidence", 0)  # ❌ Why 0?
 
# Line 126
adx = detailed.get("adx", {}).get("current", 25.0)  # ❌ Why 25.0?
 
# Line 127
mean_rev_strength = mean_rev.get("reversion_strength", 0.0)  # ❌ Why 0?
 
# Line 128
vol_expansion = vol_metrics.get("volatility_expansion_ratio", 1.0)  # ❌ Why 1.0?

In exit_strategy/evaluator.py:

# Line 161
confidence = regime_analysis.get("confidence", 0.0)  # ❌ Missing = no confidence?
 
# Line 162
volatility_expansion_ratio = regime_analysis.get("volatility_metrics", {}).get("volatility_expansion_ratio", 1.0)
 
# Line 163
boundary_violations = regime_analysis.get("range_analysis", {}).get("boundary_violations", 0)

In exit_strategy/entry_evaluator.py:

# Line 164
confidence = regime_analysis.get("confidence", 0.0)
 
# Line 170
range_quality = regime_scores.get("range_quality_score", 0.0)
 
# Line 172-173
vol_expansion = volatility_metrics.get("volatility_expansion_ratio", 1.0)
mean_reversion_strength = mean_reversion.get("reversion_strength", 0.0)
 
# Line 229-230
trend_strength = detailed_analysis.get("trend_strength", 0.5)  # ❌ Why 0.5?
adx = detailed_analysis.get("adx", {}).get("current", 25.0)  # ❌ Again!

3.2 Problem Analysis

These defaults mask data problems:

Silent Failures: If adx.current is missing, code uses 25.0 without warning
Inconsistent Defaults: Same field gets different defaults in different files
Arbitrary Values: Why is default ADX 25.0? Is that “neutral”? Based on what?
No Documentation: None of these defaults are explained

Impact:

Calculations proceed with wrong data
No alerts when data is missing
Different code paths get different defaults for same field
Debugging is impossible - you can’t tell real data from fake

3.3 Better Pattern (MISSING)

# ✅ SHOULD be like this:
adx_data = detailed.get("adx", {})
if "current" not in adx_data:
    logger.error("ADX data missing - cannot evaluate entry conditions")
    return EntryState.NOT_READY, ["Missing ADX data"]
 
adx_current = adx_data["current"]

4. Missing Calculations

4.1 ADX Calculation

TODO: src/regime/engine.py:293

# For now, use a reasonable default. TODO: Store ATR in detailed_analysis
atr = 1500.0  # Will be replaced when we store ATR in detailed_analysis

Evidence of calculation: src/regime/metrics/adx.py

ADX calculation EXISTS
Called in src/regime/engine.py
Result stored in detailed_analysis["adx"]

Problem: ATR is calculated but NOT stored consistently

Sometimes stored in volatility_metrics.current_atr
Sometimes stored in detailed_analysis.adx.atr
Sometimes not stored at all (uses hardcoded 1500.0)

4.2 Baseline Calculations (CRITICAL MISSING)

TODO Locations:

# src/exit_strategy/evaluator.py:362
# TODO: Implement 7-day rolling average during RANGE_OK periods.
 
# src/exit_strategy/evaluator.py:385
# TODO: Implement 7-day rolling average during RANGE_OK periods.

Implementation Status: src/exit_strategy/evaluator.py:357-401

def _get_baseline_atr(self, symbol: str) -> Optional[float]:
    """
    Get baseline ATR for symbol.
    For now, uses configured value or calculates from history.
    TODO: Implement 7-day rolling average during RANGE_OK periods.
    """
    if self.baseline_atr:
        return self.baseline_atr  # ✅ If pre-configured
    
    # Fallback: Calculate from recent RANGE_OK history
    try:
        history = self.history_loader.load_recent_metrics(symbol, hours=168)  # 7 days
        range_ok_metrics = [m for m in history if m.get("regime_verdict") == "RANGE_OK"]
        if range_ok_metrics:
            atr_values = [m["atr"] for m in range_ok_metrics if m.get("atr") is not None]
            if atr_values:
                return sum(atr_values) / len(atr_values)  # ⚠️ Simple average, not rolling
    except Exception as e:
        logger.warning(f"Failed to calculate baseline ATR for {symbol}: {e}")
    
    return None  # ❌ Returns None if no baseline available

Problem:

Fallback only: Baseline calculation is a fallback, not the primary method
Simple average: Not a rolling average, just averages all RANGE_OK periods
No storage: Calculated baseline is NOT stored for reuse
No updates: Baseline never updates as new data arrives

Impact:

Exit triggers that compare current ATR to baseline ATR fail
check_volatility_expansion() gets None for baseline
Trigger evaluation skips volatility checks entirely

4.3 Grid Performance Metrics

Available but NOT Used:

In YAML files:

analysis:
  grid_analysis:
    grid_ladder:
      - level: 7
        status: "SELL"
        buy_price: 320000  # ❌ Integer!
        sell_price: 325000  # ❌ Integer!

Available in GridConfigurationManager:

Grid bounds (upper/lower)
Grid spacing
Amount per grid
Grid history (enabled/disabled timestamps)

NOT Calculated:

How many levels were hit this hour?
What was the actual profit per grid?
How does spacing compare to volatility?
Is the grid optimally positioned?

Result: Rich grid data exists but goes unused in decision-making

5. Notification Generation Logic

5.1 How Notifications Are Created

Entry Point: send_regime_notifications.py:main() → Lines 498-645

Data Flow:

Load latest metrics YAML → load_latest_metrics()
Extract regime analysis from metrics["analysis"]["regime_analysis"]
Evaluate exit/entry state → ExitStateEvaluator.evaluate() or GridEntryEvaluator.evaluate()
Generate recommendations → generate_entry_recommendations() or use exit_reasons
Send notification → send_pushover_notification()

5.2 Which Fields Are Used

From regime_analysis:

verdict = regime.get("verdict", "UNKNOWN")  # ✅ Used
confidence = regime.get("confidence", 0.0)  # ✅ Used

From range_analysis:

range_conf = range_analysis.get("range_confidence", 0)  # ⚠️ Used in one place

From volatility_metrics:

vol_expansion = vol_metrics.get("volatility_expansion_ratio", 1.0)  # ⚠️ Used conditionally
vol_state = vol_metrics.get("volatility_state", "UNKNOWN")  # ⚠️ Used for display

From mean_reversion:

rev_strength = mean_rev.get("reversion_strength", 0)  # ⚠️ Used conditionally

5.3 Which Fields Are IGNORED

regime_scores - COMPLETELY IGNORED:

trend_score - NOT used in notifications
mean_rev_score - NOT used in notifications
vol_level_score - NOT used in notifications
vol_change_score - NOT used in notifications
range_quality_score - NOT used in notifications
grid_capacity - NOT used in notifications

detailed_analysis - PARTIALLY IGNORED:

adx - Only used in entry evaluation for directional emergence check
efficiency_ratio - NOT used
slope - NOT used
autocorrelation - NOT used
ou_process - NOT used
bollinger - NOT used

grid_analysis - COMPLETELY IGNORED:

grid_ladder - NOT used
grid_performance - NOT used

5.4 What Makes Notifications “Static”

Hard-coded Logic: send_regime_notifications.py:108-230

if verdict == "RANGE_OK":
    recs.append("Range conditions favorable for grid trading")
    if range_conf >= 0.8:  # ❌ Hard-coded threshold
        recs.append(f"Strong range detected (confidence: {range_conf:.2f})")
 
elif verdict == "RANGE_WEAK":
    # ❌ Hard-coded multi-condition check
    checks = {
        "trend_strength > 0.4": (trend_strength > 0.4, trend_strength, 0.4),
        "adx > 25": (adx > 25, adx, 25),
        "mean_rev < 0.3": (mean_rev_strength < 0.3, mean_rev_strength, 0.3),
        "vol_expansion > 1.2": (vol_expansion > 1.2, vol_expansion, 1.2),
    }

Result:

Recommendations are based on verdict + 3-4 specific metrics
All the detailed calculations (regime_scores, feature ranks, etc.) are thrown away
Notification text is template-based, not data-driven

6. Data Flow Diagram

6.1 Collection → Storage → Analysis → Notification

┌─────────────────────────────────────────────────────────────────┐
│ 1. COLLECTION (metrics-service/collect_metrics.py)             │
├─────────────────────────────────────────────────────────────────┤
│ Exchange API → Price Data (OHLCV)                              │
│             → Grid Status (from history)                        │
│             → Account Balance (if recent)                       │
│                                                                 │
│ Calculations:                                                   │
│   • Regime analysis → RegimeEngine.get_current_regime()        │
│   • Feature calculation → FeatureCalculator.calculate_raw()    │
│   • Score aggregation → ScoreAggregator.aggregate_all()        │
│   • Grid comparison → compare_discovered_vs_configured()       │
│                                                                 │
│ ❌ BUG: Integer conversion here:                                │
│   prices * 100 → YAML                                          │
│   amounts * 10000000 → YAML                                    │
│                                                                 │
│ ❌ BUG: Fake history created:                                   │
│   adx_history = [adx] * 10                                     │
│   atr_history = [atr] * 100                                    │
└─────────────────────────────────────────────────────────────────┘
                          ↓
┌─────────────────────────────────────────────────────────────────┐
│ 2. STORAGE (metrics-service/src/metrics/history.py)            │
├─────────────────────────────────────────────────────────────────┤
│ Write YAML files:                                              │
│   metrics/YYYY/MM/DD/HH_ETH-USDT.yaml                         │
│                                                                 │
│ Structure:                                                      │
│   config:                                                       │
│     symbol: "ETH/USDT"                                         │
│     grid_config: {...}  ← ❌ Integer prices stored here        │
│   market_data:                                                  │
│     market_summary: {...}  ← ❌ Integer prices stored here     │
│   analysis:                                                     │
│     regime_analysis: {...}  ← ✅ Some floats, some integers    │
│       verdict: "RANGE_OK"                                      │
│       confidence: 0.75                                         │
│       regime_scores: {...}  ← ❌ Mostly ignored later          │
│       detailed_analysis: {...}  ← ❌ Partially ignored         │
└─────────────────────────────────────────────────────────────────┘
                          ↓
┌─────────────────────────────────────────────────────────────────┐
│ 3. ANALYSIS (send_regime_notifications.py)                     │
├─────────────────────────────────────────────────────────────────┤
│ Load latest YAML → load_latest_metrics()                      │
│                                                                 │
│ Extract fields:                                                │
│   verdict ✅                                                    │
│   confidence ✅                                                 │
│   range_analysis (partial) ⚠️                                  │
│   volatility_metrics (partial) ⚠️                              │
│   mean_reversion (partial) ⚠️                                  │
│                                                                 │
│ ❌ IGNORED:                                                     │
│   regime_scores (completely)                                   │
│   detailed_analysis (mostly)                                   │
│   grid_analysis (completely)                                   │
│                                                                 │
│ Exit/Entry State Evaluation:                                   │
│   → ExitStateEvaluator.evaluate()                              │
│   → GridEntryEvaluator.evaluate()                              │
│   → Uses hardcoded thresholds + .get() defaults                │
│                                                                 │
│ Generate Recommendations:                                       │
│   → generate_entry_recommendations()                           │
│   → Template-based, not data-driven                            │
└─────────────────────────────────────────────────────────────────┘
                          ↓
┌─────────────────────────────────────────────────────────────────┐
│ 4. NOTIFICATION (Pushover)                                     │
├─────────────────────────────────────────────────────────────────┤
│ Title: Based on verdict + entry/exit state                     │
│ Message: Template with 3-5 metrics                             │
│ Priority: Map from verdict/state to priority level             │
│                                                                 │
│ ❌ Integer values NOT converted back to floats                  │
│ ❌ Calculated scores NOT used                                   │
│ ❌ Static text, not adaptive                                    │
└─────────────────────────────────────────────────────────────────┘

6.2 Key Observations

One-way integer conversion: Values converted to integers during storage, never converted back
Data loss: Rich calculated data (scores, features) is stored but not used
Fake history: Historical arrays are fabricated, not loaded from past YAML files
Static logic: Decision-making uses hardcoded thresholds, not data-driven adaptation
Missing feedback loop: Baselines should update from RANGE_OK periods, but don’t

7. Systemic Issues Summary

Issue 1: Integer Conversion Bug

Severity: HIGH
Locations: 15+ write locations, 0 read conversions
Impact: All displayed prices/amounts are 100-10000000x too large
Fix Required: Add division in read paths, or remove multiplication in write paths

Issue 2: Fake Historical Data

Severity: HIGH
Locations: engine.py lines 286, 295, 299, 373, 381, 385
Impact: Gates never pass, trends never detected, confidence always wrong
Fix Required: Implement MetricsHistoryLoader integration to load real past data

Issue 3: Hardcoded Fallback Values

Severity: MEDIUM
Locations: 20+ .get() calls with arbitrary defaults
Impact: Silent failures, inconsistent behavior, impossible debugging
Fix Required: Remove defaults, add explicit error handling, log warnings

Issue 4: Missing Baseline Calculations

Severity: MEDIUM
Locations: evaluator.py lines 357-401
Impact: Exit triggers fail, volatility expansion checks skip
Fix Required: Implement 7-day rolling average, store in YAML, update periodically

Issue 5: Static Notification System

Severity: LOW
Locations: send_regime_notifications.py lines 91-230
Impact: Rich data ignored, notifications don’t adapt, users miss insights
Fix Required: Use regime_scores and detailed_analysis in recommendations

8. Recommendations

Immediate (P0)

Fix integer conversion bug:
- Add _convert_stored_integers() helper in history.py
- Call it when loading YAML in all evaluation paths
- OR remove int() multiplication in write paths
Replace fake history:
- Integrate MetricsHistoryLoader in engine.py
- Load real ADX/ATR/BB values from past YAML files
- Remove [value] * N patterns

Short-term (P1)

Implement baseline calculations:
- Add calculate_baseline_atr() and calculate_baseline_halflife()
- Store baselines in separate file (baselines.yaml)
- Update baselines daily during RANGE_OK periods
Remove hardcoded defaults:
- Replace .get(key, DEFAULT) with explicit checks
- Add logging when data is missing
- Fail fast if critical data missing

Medium-term (P2)

Use calculated scores in notifications:
- Read regime_scores from YAML
- Generate recommendations from score thresholds
- Add “diagnostic” mode showing all scores
Add grid performance metrics:
- Calculate levels hit per hour
- Measure actual vs expected profit
- Compare spacing to volatility

9. Files by Category

Core Engine Files

src/regime/engine.py - Regime classification, USES fake history, WRITES integers
src/regime/classifier.py - Hierarchical decision tree
src/regime/feature_calculation.py - Raw feature calculation
src/regime/score_aggregation.py - Aggregate features into scores

Data Storage Files

src/metrics/collector.py - Collects data, WRITES integers
src/metrics/history.py - Writes YAML files, has partial int→float converter
src/grid/configuration_manager.py - Grid config, WRITES integers

Evaluation Files

src/exit_strategy/evaluator.py - Exit state, USES hardcoded defaults, MISSING baselines
src/exit_strategy/entry_evaluator.py - Entry state, USES hardcoded defaults
send_regime_notifications.py - Notifications, IGNORES calculated scores

Supporting Files

src/exit_strategy/history_loader.py - Loads past YAML files (NOT used by engine)
src/regime/restart_gates.py - Gate evaluation (receives fake history)
src/regime/range_discovery.py - Range discovery (WRITES integers)

10. Next Steps

Prioritize fixes: Start with P0 (integer conversion + fake history)
Add tests: Write integration tests that catch these issues
Document patterns: Create coding standards to prevent recurrence
Refactor gradually: Fix one system at a time, test thoroughly

End of Deep Dive Map

Techcle Wiki

Explorer

DEEP DIVE DATA FLOW MAP