Techcle Wiki

Tracing Implementation

5 min read

HTTP Header Preservation for End-to-End Request Tracing

Project: HIP (HTTP Integration Platform)
Category: Infrastructure / Observability
Status: Planning
Created: 2026-02-07

Overview

This document specifies which HTTP headers the integration platform should preserve and pass through to enable end-to-end distributed tracing across microservices and external integrations.

Problem Statement

Without proper header propagation, it’s impossible to:

  • Trace requests across service boundaries
  • Correlate logs and metrics for a single business transaction
  • Debug issues in distributed workflows
  • Measure end-to-end latency accurately

Header Categories

1. W3C Trace Context (Primary Standard)

The W3C Trace Context is the recommended industry standard for distributed tracing.

Headers:

  • traceparent - Contains trace-id, parent-id, trace-flags
    • Format: 00-{trace-id}-{parent-id}-{trace-flags}
    • Example: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
  • tracestate - Vendor-specific trace context
    • Format: List of key-value pairs
    • Example: rojo=00f067aa0ba902b7,congo=t61rcWkgMzE

Priority: MUST implement

2. Zipkin B3 Format

Widely used in microservices architectures, especially with Spring Cloud, Istio, and Envoy.

Single Header Format (Recommended):

  • b3 - Compressed format containing all B3 fields
    • Format: {TraceId}-{SpanId}-{SamplingState}-{ParentSpanId}
    • Example: 80f198ee56343ba864fe8b2a57d3eff7-e457b5a2e4d86bd1-1-05e3ac9a4f6e3b90

Multi-Header Format (Legacy Support):

  • x-b3-traceid - 64 or 128-bit trace ID (hex-encoded)
  • x-b3-spanid - 64-bit span ID (hex-encoded)
  • x-b3-parentspanid - 64-bit parent span ID (hex-encoded)
  • x-b3-sampled - Sampling decision (1 = sampled, 0 = not sampled)
  • x-b3-flags - Debug flag (1 = debug enabled)

Priority: SHOULD implement

3. Request/Correlation IDs

Used for request tracking and correlation across systems.

Headers:

  • x-request-id - Unique identifier for the HTTP request
    • Common in Kong, NGINX, Envoy
    • Format: UUID or similar unique string
  • x-correlation-id - Business/domain-level correlation ID
    • May span multiple technical requests
    • Useful for tracking business transactions
  • request-id - Alternative format used by some systems

Priority: MUST implement

4. Cloud Provider Tracing Headers

AWS X-Ray:

  • x-amzn-trace-id - AWS X-Ray trace header
    • Format: Root={trace-id};Parent={parent-id};Sampled={0|1}
    • Example: Root=1-5759e988-bd862e3fe1be46a994272793;Parent=53995c3f42cd8ad8;Sampled=1

Google Cloud Trace:

  • x-cloud-trace-context - Google Cloud Trace header
    • Format: {trace-id}/{span-id};o={trace-options}
    • Example: 105445aa7843bc8bf206b120001000/0;o=1

Azure Application Insights:

  • request-context - Azure correlation context
  • request-id - Azure Application Insights request ID

Priority: SHOULD implement (based on deployment environment)

5. Service Mesh Headers

Istio/Envoy:

  • x-envoy-attempt-count - Number of retry attempts
  • x-envoy-external-address - External client address
  • x-envoy-original-path - Original request path before rewrite
  • x-envoy-decorator-operation - Operation name for tracing
  • x-forwarded-for - Client IP address chain
  • x-forwarded-proto - Original protocol (http/https)
  • x-forwarded-host - Original host header

OpenTracing:

  • x-ot-span-context - OpenTracing span context

Priority: MAY implement (if using service mesh)

Implementation Strategy

Phase 1: Core Tracing (Must Have)

  1. W3C Trace Context Support

    • Read and preserve traceparent and tracestate
    • Generate new traceparent if not present
    • Propagate to all downstream HTTP calls

  2. Request ID Support

    • Read and preserve x-request-id and x-correlation-id
    • Generate x-request-id if not present
    • Add platform-specific x-hip-request-id for internal tracking

  3. Logging Integration

    • Extract trace/span/request IDs
    • Include in structured logs
    • Use for log correlation

Phase 2: B3 Support (Should Have)

  1. B3 Single Header

    • Support reading b3 header
    • Propagate downstream
    • Convert between W3C and B3 formats if needed

  2. B3 Multi-Header (Legacy)

    • Support reading legacy x-b3-* headers
    • Preserve for backward compatibility

Phase 3: Cloud Provider Support (Nice to Have)

  1. Environment Detection

    • Detect if running in AWS/GCP/Azure
    • Enable appropriate headers automatically

  2. Cloud-Specific Headers

    • Preserve cloud provider tracing headers
    • Integrate with cloud tracing services

Header Propagation Rules

Inbound Requests

  1. Extract all tracing headers from incoming request
  2. Validate header formats (fail gracefully on invalid)
  3. Generate missing required headers:
    • If no traceparent: Generate new trace ID
    • If no x-request-id: Generate new request ID
  4. Store in request context for the workflow execution

Outbound Requests

  1. Inject all preserved headers into downstream HTTP calls
  2. Update span context:
    • Keep same trace ID
    • Generate new span ID for the outbound call
    • Set parent span ID to current span ID
  3. Add platform-specific headers:
    • x-hip-request-id: Internal platform request ID
    • x-hip-workflow-id: Workflow execution ID
    • x-hip-node-id: Current node in workflow

Workflow Context

Headers should be accessible:

  • In workflow variables/context
  • For conditional logic (e.g., if sampled, log more details)
  • For custom nodes that need tracing info

Configuration

Provide configuration options:

tracing:
  enabled: true
  
  # Header preservation
  preserve_headers:
    - traceparent
    - tracestate
    - x-request-id
    - x-correlation-id
    - b3
    - x-b3-*
    - x-amzn-trace-id
    - x-cloud-trace-context
  
  # Auto-generation
  generate_if_missing:
    traceparent: true
    x-request-id: true
  
  # Platform headers
  add_platform_headers:
    x-hip-request-id: true
    x-hip-workflow-id: true
    x-hip-node-id: true
  
  # Sampling
  sampling_rate: 1.0  # 0.0 to 1.0 (1.0 = 100%)
  
  # Integration
  export_to:
    - jaeger
    - otlp
    - datadog

Security Considerations

  1. Header Size Limits

    • Enforce max size for tracestate (512 characters recommended)
    • Total header size limits

  2. Sanitization

    • Validate header formats to prevent injection attacks
    • Strip invalid characters

  3. Sensitive Data

    • Never include sensitive data in trace headers
    • Be cautious with custom headers

  4. Cross-Tenant Isolation

    • Ensure trace IDs don’t leak between tenants
    • Consider tenant-specific trace contexts

Testing Strategy

  1. Unit Tests

    • Header parsing and generation
    • Format validation
    • Context propagation

  2. Integration Tests

    • End-to-end trace through workflow
    • Multi-service trace correlation
    • Header preservation across nodes

  3. Load Tests

    • Performance impact of header processing
    • Memory usage with large tracestate

Observability Integration

Logging

All logs should include:

{
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "00f067aa0ba902b7",
  "request_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "correlation_id": "business-txn-12345"
}

Metrics

Tag metrics with:

  • trace_sampled: Whether request is traced
  • has_trace_id: Whether trace ID was present

Distributed Tracing

Export spans to:

  • Jaeger
  • Zipkin
  • OpenTelemetry Collector
  • Datadog APM
  • New Relic
  • Cloud provider tracing (X-Ray, Cloud Trace, etc.)

References

Next Steps

  1. Review with platform team
  2. Prioritize implementation phases
  3. Create implementation tasks
  4. Define acceptance criteria
  5. Plan integration with existing observability stack

Open Questions

  • What tracing backend(s) are we using? (Jaeger, Datadog, etc.)
  • Are we running in a service mesh? (Istio, Linkerd, etc.)
  • What cloud provider(s) are we deployed on?
  • Do we need custom domain-specific correlation headers?
  • What’s the current observability stack?
  • Performance requirements for header processing?

Graph View