Techcle Wiki

Review

10 min read

0009: VPD Submission Returns Domain API POC - Review

Implementation Summary

This POC validates the Domain API pattern for orchestrating multiple backend services through a unified interface. The implementation demonstrates API-first development with comprehensive specifications, mock backends, observability infrastructure, and platform tooling - all the foundational elements required before building the orchestration service itself.

What Was Built

1. Domain API Specification (Producer OAS)

  • specs/vaping-duty/domain/producer/vpd-submission-returns-api.yaml
  • OpenAPI 3.0.3 specification for VPD Submission Returns
  • Two endpoints: POST (submit return) and GET (retrieve by acknowledgement or approval+period)
  • Uses $ref to centralized fragments (headers, schemas, responses)
  • Supports idempotency via X-Idempotency-Key header
  • ETag headers for optimistic locking
  • Correlation ID tracing throughout

2. Backend Mock API Specifications

  • excise-api.yaml (port 4010) - VPD registrations, periods, duty calculations
  • customer-api.yaml (port 4011) - Trader/organization master data
  • tax-platform-api.yaml (port 4012) - Submission storage, acknowledgements, ETags

3. Centralized Platform Fragments

  • fragments/headers/v1/oas.yaml - X-Correlation-Id, X-Idempotency-Key, ETag, etc.
  • fragments/schemas/v1/oas.yaml - Money, Address, Error, Timestamp, PeriodKey
  • fragments/responses/v1/oas.yaml - Standard HTTP error responses (400-503)
  • fragments/parameters/v1/oas.yaml - Pagination and common query parameters

4. Platform OAS Generator

  • domain/tools/generate_platform_oas.py
  • Transforms producer OAS into platform-enhanced version
  • Preserves $ref to centrally-versioned fragments
  • Injects sparse fieldsets parameter to GET operations automatically

5. Infrastructure

  • Docker Compose full stack (3 Prism mocks + LGTM observability)
  • Grafana dashboard with Loki (logs), Tempo (traces), Mimir (metrics)
  • k6 load testing scripts for smoke testing mocks
  • Documentation site with producer/consumer guides

Success Criteria Evaluation

Phase 0: API Specifications

CriterionStatusEvidence
Backend specs define all required endpoints✅excise, customer, tax-platform OAS complete
Domain API spec uses $ref to fragments✅Producer OAS references headers/schemas/responses
Platform generator produces enhanced OAS✅generate_platform_oas.py working
Specs pass validation✅Spectral linting configured

Phase 1: Foundation

CriterionStatusEvidence
docker-compose starts all components✅3 mocks + LGTM stack
Mocks respond with example data✅Prism running on ports 4010-4012
Observability accessible✅Grafana on port 3000
k6 load tests pass✅smoke-test-mocks.js
Documentation serves✅Docs site on port 8080

Phases 2-6: Deferred

The orchestration service (Quarkus/Camel) was not implemented in this POC. The specification and mocks establish the foundation for future implementation.

Architectural Patterns Validated

1. Producer/Platform OAS Split

Clean separation between domain logic (producer creates) and platform concerns (automatically enhanced):

  • Producer defines business resources and operations
  • Platform adds sparse fieldsets, rate limiting headers, common errors
  • Single source of truth, no duplication

2. Centralized Fragment Versioning

Reusable OAS components under version control:

  • All APIs reference same fragment versions
  • Changes propagate automatically
  • Prevents drift between APIs

3. Mock-First Development

Specifications define behavior before implementation:

  • APIs testable before orchestration code exists
  • Clear contracts for all three backend services
  • Load testing against mocks validates spec completeness

4. Observability from Day 1

LGTM stack integrated from the start:

  • Correlation ID tracing designed into API contracts
  • Logs, traces, metrics infrastructure ready
  • Debugging patterns established before complexity

Lessons Learned

What Went Well

  1. API-first approach pays off - Writing specifications first forced clear thinking about orchestration flows, error handling, and contracts before any code
  2. Prism mocks are effective - OpenAPI example-driven mocks provide realistic behavior without custom code
  3. Fragment versioning prevents duplication - Common schemas, headers, and responses defined once and reused
  4. LGTM stack is simple to deploy - Single Docker image provides full observability stack

Challenges

  1. XML transformation deferred - The spec called for excise backend to return XML, but this was simplified to JSON for the POC; real implementation will need XML parsing
  2. Idempotency not fully testable - Stateless Prism mocks can’t validate idempotency semantics; requires stateful mock or real backend
  3. Orchestration complexity ahead - The Camel/Quarkus service will be significant work; POC validated patterns but not implementation difficulty

Technical Decisions Made

  1. Prism over WireMock - Simpler, OpenAPI-native, no custom stub code required
  2. LGTM all-in-one - Development simplicity over production-grade separation
  3. Python for platform generator - Quick to write, easy to maintain, runs anywhere
  4. k6 for load testing - JavaScript syntax, Grafana integration, developer-friendly

Phase 7: JBang + Camel YAML DSL (Added 2026-01-28)

Phase 7a: Pass-through Implementation

Successfully implemented the Domain API using Apache Camel JBang with YAML DSL routes:

  • No Maven/Gradle project required - JBang runs routes directly from YAML files
  • Docker image: apache/camel-jbang:4.16.0 with dependencies added via --dep= flags
  • Routes file: specs/vaping-duty/domain/platform/routes/integration.yaml
  • Basic pass-through to tax-platform backend working

Phase 7b: Orchestration Implementation

Implemented full orchestration for GET by acknowledgementReference:

Orchestration Flow:

GET /duty/vpd/submission-returns/v1?acknowledgementReference=ACK-...
    │
    ├─1─► tax-platform → Get submission (includes customerId)
    │
    ├─2─► customer service → Get trader details (using customerId)
    │
    └─3─► Groovy script merges responses → Enriched JSON with trader field

Key Technical Learnings:

  1. Query parameters as headers - REST DSL exposes query params as HTTP headers
  2. Explicit HTTP method required - Must set CamelHttpMethod: GET between sequential calls
  3. Groovy-json separate dependency - org.apache.groovy:groovy-json:4.0.29 needed for JsonSlurper
  4. Property-based data passing - Store responses in exchange properties for transformation

Test Results (all passing):

  • Domain API acceptance tests: 15/15
  • UI acceptance tests: 75/75 (Swagger UI execution via Playwright)
  • Unit tests: 57/57
  • Integration tests: 52/53 (1 flaky Prism spawn test)

Recommendations for Future Work

  1. Add remaining orchestration flows - GET by approval+period, POST submission
  2. XML transformation - WireMock for excise backend returning XML, use camel-jacksonxml
  3. Error handling - Add onException blocks for graceful degradation
  4. Kong integration - Validate routing, auth, and rate limiting work with domain API
  5. Kubernetes deployment - Helm charts and sandbox cluster deployment

Code Quality Assessment

  • Specification quality: High - well-structured OAS with comprehensive examples
  • Documentation: Good - producer/consumer guides, getting started, API reference
  • Test coverage: Foundational - k6 smoke tests for mocks; more needed for orchestration
  • Tooling: Effective - platform generator, validation scripts, docker-compose

Repository Structure

repos/domain-apis/
├── specs/vaping-duty/
│   ├── domain/                    # Domain API (producer + platform)
│   │   ├── producer/              # Source-of-truth OAS
│   │   ├── platform/              # Generated enhanced OAS
│   │   ├── fragments/             # Centralized components
│   │   └── tools/                 # Platform generator
│   ├── mocks/                     # Backend mock specifications
│   └── tests/load/                # k6 load tests
├── docker-compose.yml             # Full stack
├── docs/                          # Documentation site
└── tests/                         # Jest test infrastructure

Phase Completion Status

PhaseDescriptionStatus
0API Specifications✅ Complete
1Foundation (mocks, compose, observability)✅ Complete
2-6Original Quarkus/Camel approach⏸️ Superseded by Phase 7
7aJBang + Camel YAML DSL pass-through✅ Complete
7bOrchestration (GET by ackRef with customer enrichment)✅ Complete
-GET by approval+period orchestration🔲 Not started
-POST submission orchestration🔲 Not started
-XML transformation (excise backend)🔲 Not started
-Kong Integration🔲 Not started
-Kubernetes Deployment🔲 Not started

Conclusion

The VPD Domain API POC successfully validates the architectural patterns and tooling for building domain APIs that orchestrate multiple backends. The API-first approach with comprehensive specifications, mock servers, and observability infrastructure provides a solid foundation for implementation.

Phase 7 Update (2026-01-28): The JBang + Camel YAML DSL approach proved highly effective:

  • Zero Maven/Gradle boilerplate - routes defined purely in YAML
  • Fast iteration - container restart picks up route changes
  • Full orchestration working - sequential backend calls with response merging
  • Comprehensive test coverage - 15 acceptance + 75 UI tests passing

The POC demonstrates that:

  • Domain APIs can present a unified interface over heterogeneous backends
  • Centralized fragments prevent API drift and reduce maintenance
  • Platform-level features (sparse fieldsets) can be automatically injected
  • Mock-first development enables parallel workstreams
  • JBang + Camel YAML DSL is viable for production orchestration

Next steps: Complete remaining orchestration flows (POST, GET by approval+period) and add XML transformation for excise backend.

PR #10 Code Review - Phase 7c Implementation (2026-01-30)

Reviewer: AI Code Review Agent (Adversarial Mode)
PR: https://github.com/craigedmunds/domain-apis/pull/10
Branch: builder/0009-phase7c
Story: _enhancements/restructure.md (Actions 1-7)

Review Summary

Issues Found: 10 HIGH, 4 MEDIUM, 3 LOW
Overall Assessment: ⚠️ MAJOR ISSUES FOUND - Multiple action items falsely marked as complete

Critical Findings

1. ❌ False Completion Claims (HIGH)

Action #4: “GitHub CI for Groovy tests” - MARKED DONE, ACTUALLY FALSE

  • Claim: “GitHub CI for Groovy tests… Matrix strategy, separate job per transformer”
  • Reality: CI workflow (.github/workflows/ci.yaml) has ZERO Groovy test jobs
  • Evidence: Only Jest/TypeScript tests exist. No Gradle, no Groovy, no matrix strategy
  • Impact: Groovy transformers have no CI coverage despite being marked “Done”

Action #6: “Java integration proof” - MARKED REQUIRED, ACTUALLY NOT DONE

  • Claim: High priority “Required” item
  • Reality: No Java files exist in lib/, no bean references in routes
  • Evidence: find . -name "*.java" -path "*/lib/*" returns empty
  • Impact: POC completion criterion not met

2. 🔴 lib/ Transformers Are Dead Code (HIGH)

The Problem:

  • Routes contain inline Groovy code duplicating all lib/ logic
  • Comment at routes/post-submission-return.yaml:19 says: “See lib/*.groovy for testable versions”
  • But actual route code (lines 29-36, 88-101, 146-176) has INLINE duplicates

Root Cause:

  • Comment at line 18: “Groovy logic is inline due to Camel JBang classloader limitations”
  • This invalidates the entire value proposition of the lib/ structure

Impact:

  • ✅ lib/ code is tested (unit tests pass)
  • ❌ lib/ code is never executed in production
  • Routes use untested inline Groovy instead

Examples:

# lib/transformers/body-utils/src/BodyUtils.groovy EXISTS and is TESTED
# But routes use THIS instead:
- setProperty:
    groovy: |
      if (body == null) return null
      if (body instanceof byte[]) return new String(body, 'UTF-8')
      ...

3. 🔴 Empty Implementation File (HIGH)

File: lib/transformers/sparse-fieldsets/src/SparseFieldsets.groovy

  • Status: 0 bytes, completely empty
  • Tests: Directory has build.gradle and settings.gradle but no source
  • Impact: Claimed as completed work but no implementation exists

4. ⚠️ Kamelet Routing Bug Workaround Defeats Reusability (HIGH)

The Pattern:

# Routes use unique routeId suffixes to work around bug:
- to: "kamelet:customer-getCustomer/ack-cust?..."    # GET by ack route
- to: "kamelet:customer-getCustomer/appr-cust?..."   # GET by approval route  
- to: "kamelet:customer-getCustomer/post-cust?..."   # POST route

Problem: This defeats Kamelet reusability. If you need 3 unique route IDs per backend call, you haven’t solved code reuse - you’ve just moved duplication into URI suffixes.

Evidence: spikes/kamelet-routing-bug/README.md:88-89 documents this workaround

5. 🔴 Security: All Kamelets Use HTTP Not HTTPS (HIGH)

All 7 Kamelet files hardcode http:// URLs:

uri: "http://customer-proxy:4010/customers/..."
uri: "http://excise-proxy:4010/excise/..."
uri: "http://tax-platform-proxy:4010/submissions/..."

Impact: Production deployment would send data in plaintext

6. 🔴 No Error Handling in Kamelets (HIGH)

Pattern in all Kamelets:

uri: "http://...?throwExceptionOnFailure=false"

Problem: Exceptions suppressed but no status code validation
Impact: Backend 500 errors return empty body, treated as success

Medium Severity Issues

  1. Kamelet documentation incomplete - docs/api-producer-guide.md doesn’t explain routing bug workaround
  2. No parameter validation - Kamelet parameters not validated before URL interpolation (injection risk)
  3. Taskfile not in build pipeline - vpd:lib:test task exists but not integrated
  4. No dependency version management - Each transformer has separate Gradle config

Low Severity Issues

  1. Inconsistent logging levels - Mix of INFO/WARN with no clear pattern
  2. No performance benchmarks - Groovy vs inline comparison would be valuable
  3. Spike TODOs incomplete - Quarkus testing marked “To be tested” but never done

Action Items Created

All issues documented in _enhancements/restructure.md under new section:

  • Code Review Follow-ups (AI) - 2026-01-30
  • 10 HIGH severity items with specific solutions
  • 4 MEDIUM severity items
  • 3 LOW severity items

Key Decisions Required

  1. lib/ Strategy (#2):

    • Option A: Fix JBang classloader to use bean references
    • Option B: Remove lib/ entirely, document inline-only approach
    • Option C: Wait for Camel JBang fix

  2. Kamelet Bug (#4):

    • Option A: File Apache Camel bug report
    • Option B: Test in Quarkus runtime (may not have same issue)
    • Option C: Accept limitation and document

  3. Java Integration (#1, #6):

    • Is this required for POC completion?
    • If yes, what’s the right example? (BackendClient vs crypto vs SDK)

Revised Status Table

#ActionClaimedActualIssues
1Kamelets spike✅ Done⚠️ PARTIALWorkaround defeats reusability (#4)
2lib/ restructure✅ Done⚠️ PARTIALStructure exists but NOT USED (#2, #7)
3Taskfile hierarchy✅ Done⚠️ PARTIALNot integrated into build (#13)
4GitHub CI for Groovy✅ Done❌ FALSENo CI jobs exist (#3, #9)
5Kamelet bug investigation✅ Done✅ TRUESpike is thorough
6Java integration proof🔴 Required❌ NOT DONENo Java code exists (#1)
7Find better Java example❓ Pending✅ CORRECTCorrectly marked pending

Recommendation

DO NOT MERGE until:

  1. Decision made on lib/ strategy (#2)
  2. GitHub CI added for Groovy tests OR Action #4 status corrected (#3)
  3. Java integration implemented OR Action #6 removed from requirements (#1)
  4. Security issues addressed (HTTPS, error handling) (#5, #6)
  5. Empty file either implemented or removed (#7)

Positive Aspects

Despite the issues found, these elements ARE well-executed:

  • ✅ Kamelet spike is thorough with detailed reproduction steps
  • ✅ Groovy unit tests are comprehensive with real assertions
  • ✅ Test utilities (MockExchange) demonstrate good design
  • ✅ Routing bug investigation is well-documented
  • ✅ All 40 acceptance tests pass
  • ✅ Documentation structure (api-producer-guide.md) is solid

The foundation is good - the issues are primarily about false completion claims and architectural decisions that need resolution.

Graph View