Techcle Wiki

Initial Thoughts

4 min read

We are an integration platform in the UK government, extending our “Simple API” feature to include camel yaml steps.

Currently an API producer uploads their OAS file and selects a destination egress. The request is passed almost as-is to that destination when a request is received that matches the defined route.

The new mechanism, after uploading their OAS file, will be able to define a pipeline for each route in the OAS, visually.

The main blocks will be:

  • Call API; abstractions over APIs that exist in the catalogue (and which will handle concerns like auth, content type, formatting, etc
  • Content transformation including field renaming
  • Choice (ie if, else if, else)
  • For loops to iterate based on a list (ie a field with an array, or list of objects)
  • Parrallel execution; forking to 2 or more pipelines, sync (waits for all to complete) or async (fire and forget)
  • Script; adding some groovy code

Each step will allow the user to select fields to pass in, selecting fields from the current context (which would include the incoming request as defined in the oas for the route, as well as any previous step in the pipeline) to map to that step, for example to map fields to the inputs of an API to call.

We have an example oas (vpd-submission-returns-api.yaml) and integration (post-submission-return.yaml) alongside this file.

Design Decisions

TODO:

  • how do we do versioning of integration, deployment through envs, versioning of oas
  • Do we call require apis to exist in hub? Do we allow oas/wsdl to be uploaded inline? Do we publish them to the catalogue if so?
  • How/do we need to handle hard filtering needs of the called apis?
  • How does test work? User will need creds with appropriate scopes? Can create/update an application for the purpose?
  • Arrays/lists
  • Show inputs and outputs of each stage when test is clicked? Also http requests of call?
  • Combine monitoring/tracing?
  • Allow test cases/results to be managed
  • Allow nfrs to be defined on the pipeline
  • Display slas on the api calls : expected response time, p50,p99, max throughput supported, max available

Data & Context Management

  • Context representation: Dropdown of source (original request, step xxx, etc) followed by a linked dropdown of fields from that source
  • Field selection: Dot notation for nested JSON structures (e.g., submission.basicInformation.submittedBy.name)
  • Type awareness: Maintain data type for each field; infer safe conversions where possible; use Groovy stage for explicit type conversion when needed
  • Variable naming: Step outputs are named as the source (clarification needed: “As in the source?“)

Pipeline Execution Model

  • Error handling: Yes, include try/catch blocks or error routing
  • Conditional execution: Keep it simple - dropdown with basic operators (equals, not equals, greater than, etc.) and static value or pick from data
  • Loop context: Current item appears in the source dropdown
  • Parallel execution sync: Results from parallel branches appear as items in the source list

API Catalogue Integration

  • API discovery: Dropdown list with API ID in the name; smart search capability; link to external catalogue (opens in new window/tab)
  • Request/response mapping: List the inputs with a mapping dropdown for each

Visual Design

  • Canvas layout: Top-to-bottom flowchart (open to suggestions)
  • Step configuration: Inline editing (expand steps inline)
  • Context visibility: Dropdowns, possibly with sidebar if helpful
  • Validation feedback: Real-time validation as they build

Technical Implementation

  • Testing: Yes, provide ability to test pipeline before deploying (mock data, sandbox environment)
  • Groovy support: Users can break out to Groovy if needed (not required in initial example)
  • Kamelet URIs: Abstracted away from the user
  • Property/header/body: UI shows whether something is a header or in the body, but user doesn’t need to manage this directly

Future Complexity (Out of Scope for Now)

These items are acknowledged but deferred for future iterations:

Data & Context

  • Deeply nested arrays and complex nested structures
  • Advanced type conversion scenarios

Conditional Expressions

  • Full expression language support (JSONPath, complex Groovy expressions)
  • Advanced conditional logic beyond simple operators

API Catalogue

  • Auth configuration management
  • API versioning handling

Technical Implementation

  • Camel YAML generation strategy (direct mapping vs intermediate representation)
  • Production debugging and tracing
  • Version control integration for pipeline definitions

Graph View