Pipeline specifications

A pipeline specification is the reviewable YAML or JSON definition of a pipeline. It names the steps and their data contracts, but never contains connection credentials. Git is the source of truth. When you apply a commit, Braidplane stores an immutable version for validation and deployment.

Use filenames ending in .pipeline.yaml, .pipeline.yml, or .pipeline.json. The examples below use YAML.

Top-level shape

metadata:
  slug: customer-export
  name: Customer export
steps:
  - kind: query
    slug: read-customers
    query:
      data_source_ref: customers-db
      operation_type: sql
      operation_ref: active-customers
      output_contract:
        representation: rows
        fields:
          - name: customer_id
            type: string
            required: true

metadata.slug and every step slug use lowercase letters, digits, and hyphens. Slugs are stable identifiers; names are display labels. steps must contain at least one step and form an acyclic graph.

Step kinds

Kind	Purpose	Required block
input	Accept one run-scoped JSON payload	input.output_contract
query	Read from a connector	query
transform	Transform earlier results with sql_v1	transform
sink	Write one earlier result to a connector	sink

Query steps are graph roots. A transform uses a map of SQL table aliases to earlier step slugs. A sink consumes exactly one earlier step:

- kind: transform
  slug: select-customer-ids
  transform:
    inputs:
      customers: read-customers
    sql: SELECT customer_id FROM customers
    output_kind: rows

- kind: sink
  slug: write-customers
  sink:
    input: select-customer-ids
    data_source_ref: archive
    operation_type: s3
    operation_ref: write-customer-export
    idempotency_class: naturally_idempotent

This example assumes the saved S3 operation overwrites a stable object key. Choose another replay class if the operation does not satisfy that guarantee.

Every transform or sink reference must point to an earlier input, query, or transform step. Query-parameter step references must point to an earlier query or transform step. Multiple query roots are allowed when they converge downstream.

Connections and operations

data_source_ref is a logical name, not the slug of a saved connection. Bind each logical reference to a compatible data source separately for every project environment. That keeps one specification portable across runtime targets.

Set operation_type to the connector operation family:

• Query: sql, bigquery, snowflake, http, hubspot, ftp, mongodb, git, or s3
• Sink: sql, bigquery, snowflake, http, ftp, mongodb, git, or s3

Define exactly one operation target:

• operation_ref: slug of a saved connector operation; recommended when the operation is managed and tested independently.
• operation_inline: connector-specific statement embedded in the spec.

Connector guides define the accepted statement fields.

Parameters and {{placeholders}}

Use {{name}} in a connector operation when part of the operation must receive a value at execution time. Names are case-sensitive. For example, a saved SQL query might contain:

SELECT customer_id, email
FROM customers
WHERE customer_id = {{customer_id}}

A saved query defines each ordinary parameter's name, type, whether it is required, and any default. Supported types are text, integer, boolean, date, and uuid. The pipeline query step supplies the value under the same name. An inline operation has no saved parameter definition, so put every ordinary placeholder value in the query step's parameters map and use the connector's supported value types. Reserved {{runtime.*}} expressions may also appear directly in supported inline-operation fields.

- kind: query
  slug: read-customer
  query:
    data_source_ref: customers-db
    operation_type: sql
    operation_ref: customer-by-id
    parameters:
      customer_id: "{{runtime.input.customer_id}}"

Here, {{customer_id}} belongs to the saved operation, while {{runtime.input.customer_id}} tells the pipeline where its value comes from. Start the run with a non-secret input value:

braidplane pipeline start customer-export \
  --input-json '{"customer_id": 42}'

A query parameter value can be:

• a literal JSON-compatible value, such as customer_id: 42;
• a per-run non-secret value, {{runtime.input.customer_id}};
• an environment value, {{runtime.env.batch_size}};
• a write-only environment secret, {{runtime.secrets.API_TOKEN}}; or
• an earlier query or transform result, such as $steps.read-customers.rows.

The runtime.* namespaces are reserved and supplied by Braidplane. {{input}} is also reserved for the upstream payload consumed by a sink; it is not an ordinary query parameter. Keep connection credentials on the data source, not in pipeline literals or per-run input.

Do not quote {{name}} inside SQL or use it for table or column names. SQL connectors bind it as a typed value. Other connectors substitute parameters only in the fields documented by their connector guide. A missing required value or a value that cannot be converted to its declared type stops the operation with a validation or execution error.

SQL transforms

sql_v1 runs one SELECT over named, row-normalized inputs. It supports projections, aliases, comparisons and boolean filters, IS NULL, grouped aggregates, ORDER BY, literal LIMIT/OFFSET, non-recursive CTEs, and JSON_OBJECT(...) or JSON_ARRAY(...) projection helpers.

It rejects writes, multiple statements, recursive CTEs, unsupported subqueries, HAVING, and expressions outside the validated subset. Input aliases must match the names under transform.inputs; derived projections should have explicit aliases. Set output_kind to rows, json, or csv to control serialization. Local validation reports unsupported syntax before the spec is committed.

Contracts

Contracts describe the payload crossing a step boundary. Declare an upstream output_contract, a downstream input_contract, or both when you want both sides documented and checked.

output_contract:
  name: customers
  representation: rows
  fields:
    - name: customer_id
      type: string
      required: true
      validations:
        min_length: 1
    - name: created_at
      type: datetime
      nullable: true

Representations are rows, json, csv, file, and multipart_form. Transforms support rows, json, and csv; multipart form is limited to HTTP boundaries. Field types are string, integer, number, boolean, object, array, and datetime.

Fields can set required, nullable, and description. Applicable validation rules include enum, min, max, min_length, max_length, format (email, uuid, or url), and unique. Validation rejects incompatible contracts before a run starts.

Sink replay safety

Every sink declares what Braidplane may do when a write might have happened but the run did not finish cleanly. This matters after a timeout or worker crash: a retry could otherwise write the same records twice.

Class	What it means on retry
naturally_idempotent	Repeating the exact write has the same visible result. A deterministic file overwrite or an upsert by a stable business key can qualify; a plain insert does not.
idempotent_with_key	Braidplane sends the same stable idempotency key again, and the provider guarantees that the repeated key does not create another effect.
deduplicated_by_ledger	Braidplane records the attempt before calling the provider. It skips a recorded success, but refuses to guess when an attempt is still in progress because the remote outcome is unknown.
unsafe	There is no repeat-safety guarantee. After an attempt starts, Braidplane refuses automatic retry and operator replay.

The declaration does not make an operation safe by itself. Choose a class the connector supports and that the specific operation actually satisfies. For example, the S3 walkthrough uses naturally_idempotent because it overwrites a deterministic object key. If a sink cannot be replayed safely, recover it manually and verify the destination before starting another write.

Incremental query sources

An incremental query reads from the last successfully committed position instead of reading the full source every time. The checkpoint belongs to the deployed pipeline version and source step. Braidplane commits a new checkpoint only after all dependent sinks succeed, so a failed run reads the same window again.

Use checkpoint_kind: watermark for a comparable source value such as an updated_at timestamp or increasing ID:

incremental:
  checkpoint_kind: watermark
  field: updated_at
  lookback_window: 15m
  ordering:
    - updated_at
    - id
  dedupe_key: id

The source operation must use {{runtime.incremental.lower_bound.value}} in its filter and return the configured field. Braidplane observes the greatest returned value as the next checkpoint. A lookback_window moves the lower bound back to include late records; that overlap can return records again, so use a stable ordering and make the transform or sink tolerate repeats. dedupe_key names the record identity available for that deduplication; it does not remove duplicates by itself.

Use checkpoint_kind: cursor for an opaque continuation token from an API:

incremental:
  checkpoint_kind: cursor
  cursor: next_page_token

The operation accepts {{runtime.incremental.lower_bound.cursor}} and returns the named next cursor. Braidplane stores the token without interpreting it. Cursors cannot use a lookback window.

On the first run there is no lower bound. Do not put an initial checkpoint in the specification; checkpoint seeds and backfill ranges are runtime settings. Only add incremental to a query step when its connector guide documents the checkpoint kind. PostgreSQL, MySQL, BigQuery, Snowflake, and MongoDB support watermarks; HTTP and HubSpot support cursors. FTP, Git, and S3 currently perform full reads.

Validate before committing

braidplane pipeline format --file pipelines/customer-export.pipeline.yaml --format yaml
braidplane pipeline validate --local --file pipelines/customer-export.pipeline.yaml

Local validation checks syntax, step ordering, references, operation types, contracts, transform SQL, and graph semantics without contacting Braidplane. After configuring a project and environment, server validation additionally checks bindings and cached connector schema:

braidplane pipeline validate \
  --project '<project-slug>' \
  --environment '<environment-slug>' \
  --file pipelines/customer-export.pipeline.yaml

Start from the downloadable PostgreSQL-to-S3 specification for a complete query-transform-sink example.