S3

Read and write objects in AWS S3 or an S3-compatible store. Structured objects can become pipeline rows; raw objects can pass through without parsing.

Use the shared connection workflow to create this data source in your workspace. You can read this guide without an account or provider resources. Running examples requires your own non-production system and synthetic data.

Connection settings

Field	Required	Type	Default	Notes
region	yes	string	-	AWS region or S3-compatible signing region
bucket	yes	string	-	Default bucket used by connection tests, introspection, and operations that omit bucket
endpoint	no	string	AWS S3 endpoint	S3-compatible endpoint URL for providers such as MinIO
prefix	no	string	-	Key prefix used for introspection and resolving relative operation keys
force_path_style	no	boolean	false	Required by many S3-compatible providers
timeout_seconds	no	integer	5	Request timeout in seconds (1-300)
allow_private_hosts	no	boolean	false	Requires private host access to be enabled for the deployment; hosted/default deployments reject true

Authentication and permissions

Field	Required	Notes
access_key_id	yes	S3 access key ID
secret_access_key	yes	S3 secret access key
session_token	no	Temporary-session token

Credentials belong in the connection, never in an operation statement. They are encrypted at rest and are not returned in API responses. Hosted/default deployments block loopback and private network endpoints; your laptop’s localhost is not the hosted worker’s network.

An optional session token supports temporary access-key credentials. Use provider permissions scoped to the bucket and required prefix: listing and reads for sources, intended object writes for sinks. There is no implicit workload identity, AssumeRole, or default credential-chain authentication.

Test and discover

Test access to the configured bucket and refresh metadata to list and sample objects under the prefix. Introspection pages through provider listings but exposes cached schema metadata, not a public bucket continuation API. Raw files can be listed without inferred fields.

Read example

Place a synthetic customers.csv file in the configured base path or prefix:

id,email
1,alex@example.com

This read returns one row with id and email. Adapt the path and, for Git, the branch to your connection.

{
  "type": "s3",
  "operation": "read",
  "key": "customers.csv",
  "format": "csv",
  "has_header": true
}

Write example

This direct write example stores one synthetic record as JSON. With overwrite: false, an existing target causes the write to fail. For a pipeline sink, upstream input replaces the inline content; do not rely on inline sample content being written when the pipeline supplies rows.

{
  "type": "s3",
  "operation": "write",
  "key": "exports/customers.json",
  "format": "json",
  "content": [
    {
      "id": 1,
      "email": "alex@example.com"
    }
  ],
  "overwrite": false
}

Operation reference

Field	Meaning
type	Connector operation discriminator shown in the examples
operation	read or write
key	Relative file path or object key
format	auto, csv, json, spreadsheet, or file
delimiter, has_header	CSV parsing controls
sheet, range, has_header	Worksheet and A1 range controls
content	Direct write payload; pipeline input takes precedence
bucket	Optional override of the configured bucket
content_type, metadata	Object content type and metadata values
overwrite	Whether an existing object may be replaced; choose explicitly

Use {{parameter_name}} in supported paths and worksheet/range fields. Runtime values such as {{runtime.run_date}} use UTC and are supplied by Braidplane; do not declare your own query parameters under the reserved runtime. prefix.

Pipeline and format behavior

Use operation_type: s3. Structured reads normalize CSV, JSON, and Excel workbooks to rows. format: file preserves opaque bytes for pass-through sinks. Sink operations receive upstream input; it takes precedence over saved inline content. CSV input stays raw for CSV output. JSON and spreadsheet output require structured input, so CSV-to-JSON or CSV-to-spreadsheet needs an explicit transform.

Reads are bounded full-file/full-object operations with no watermark or cursor checkpoints. Results expose row counts and fields for structured reads, write counts where available, and safe file metadata. Raw bytes are not retained in historical run diagnostics for later download.

Spreadsheet example

Given an Excel workbook with a sheet named Customers and header cells id,email, this operation reads the selected table as rows:

{
  "type": "s3",
  "operation": "read",
  "key": "customers.xlsx",
  "format": "spreadsheet",
  "sheet": "Customers",
  "range": "A1:B20",
  "has_header": true
}

Supported workbook extensions are .xlam, .xlsm, .xlsx, .xltm, and .xltx. When sheet is omitted, the first worksheet is used; worksheets are not merged. range belongs to that sheet and does not include its name. Explicit has_header: true makes the first selected row the column names. Without an explicit choice, an all-numeric first row is treated as headless with col_N column names. Empty selected cells become empty strings.

For writes, range is the top-left starting cell. Braidplane creates a new workbook; it does not preserve existing formatting, charts, macros, or template metadata and does not evaluate formulas. OpenDocument workbooks are unsupported.

Limits and troubleshooting

• No bucket management, multipart upload, presigned URLs, object version selection, custom encryption headers, storage-class, tagging, or ACL controls.
• overwrite: false is an existence check, not an atomic conditional write; concurrent writers can race.
• Relative keys resolve beneath the prefix; absolute keys and .. traversal are rejected.
• Authentication errors: check signing region, endpoint, temporary token expiry, and provider permissions. S3-compatible providers may need path-style access.
• For naturally idempotent replay, use deterministic bytes, overwrite: true, and a stable key such as runs/{{runtime.sink_replay_identity}}/output.json. Other writes need an appropriate ledger policy or explicit unsafe classification.

See operation and pipeline basics for where to use these statements, parameters, environment bindings, and replay safety. Return to the connector directory.