Test with real data
Fill a test run’s parameters from a real record instead of typing sample JSON. Pick a row from a Postgres table or an object in an S3 bucket, and Dagweave copies its fields into the sample.
You need the connector and a Production or Business plan. Test runs don’t count toward runs a month, but they do count toward a limit on runs at once if your team has one.
Pick a record
Section titled “Pick a record”- Open Test with sample data and expand Advanced: use a sample record.
- Switch to the From a connected source tab.
- Choose a source. Click Load more to see the next page of records.
- Select a record and click Use this item.
The record’s fields land in the Paste a sample tab as JSON. Fields whose names match an input parameter fill that parameter. Edit the sample if you want, then run the test.
| Source | Record fields |
|---|---|
| Postgres | The columns listed in preview_columns in the source’s Secret |
| S3 | The object’s key, size, last_modified, content_type and etag |
Dagweave never reads an S3 object’s contents. Only sources whose connector is online show up.
Set up a source
Section titled “Set up a source”You do this once per source. You need access to the connector’s namespace, and you need to be on the team that owns the connector.
1. Turn on the sources feature
Section titled “1. Turn on the sources feature”It’s off by default:
helm upgrade connector oci://ghcr.io/dagweave/charts/connector \ --namespace connector-system --reset-then-reuse-values \ --set features.sources=trueThis adds a Role that can only list Secrets, but it can list every Secret in
the connector’s namespace, since RBAC can’t filter by label. The connector reads
only the ones with a source label.
2. Register the source in Dagweave
Section titled “2. Register the source in Dagweave”Open the gear menu, Clusters, and click Add data source under Data sources. Give it a name, pick Postgres table or S3 bucket, and pick the cluster whose connector reads it. The test panel’s Add a data source link opens the same form.
The new row shows the label its Secret needs,
sources.connector.dagweave.io/id=<source id>, with a Copy label button and
the kubectl label command to run. Remove takes the source out of Dagweave
and leaves the Secret in your cluster.
To script it, send POST /api/v1/sources with an API token that has
clusters:write, or have an assistant call dagweave_create_source over
MCP. Both take the connector’s ID, a name, and a type of postgres or
s3, and answer with the same label:
{ "connectorId": "<connector id>", "name": "orders", "type": "postgres" }3. Create the Secret and label it
Section titled “3. Create the Secret and label it”In the connector’s namespace, create a Secret with any name, holding the connection details and a read-only credential. Add the label from step 2.
For Postgres:
kubectl create secret generic orders-db -n connector-system \ --from-literal=host=db.internal \ --from-literal=database=app \ --from-literal=username=dagweave_source \ --from-literal=password='...' \ --from-literal=table=orders \ --from-literal=id_column=id \ --from-literal=preview_columns=id,customer,total
kubectl label secret orders-db -n connector-system \ sources.connector.dagweave.io/id=<source id>| Key | Notes |
|---|---|
host, database, username, password |
Connection details |
port |
Defaults to 5432 |
sslmode |
Defaults to require |
table |
One table, optionally schema.table |
id_column |
The column a record is fetched by. It must also be in preview_columns |
preview_columns |
Comma-separated. These are the only columns Dagweave reads |
For S3, bucket is required. Optional keys are prefix, region, endpoint
(for S3-compatible stores; leave empty for AWS), access_key_id,
secret_access_key, session_token, insecure and addressing_style.
Scope the credential to the one table or bucket. For Postgres, SELECT on that
table. For S3, s3:ListBucket and s3:GetObject on that bucket. The metadata
read is a HEAD request, which needs s3:GetObject.
What stays where
Section titled “What stays where”Dagweave stores the source’s ID, name, type and connector, nothing more. The credential and the Secret’s name never leave your cluster.
The connector only reads:
- Postgres queries run in read-only transactions and select only the ID and preview columns. Nobody writes SQL, and the record ID is a bound parameter.
- S3 reads list objects and read metadata. Keys outside the configured prefix are rejected.
Dagweave doesn’t store the records you browse. A test run saves only the parameter values, as with a pasted sample. Fields that don’t match a parameter aren’t sent.
The audit log records every list and fetch with who, the source, the connector and the time. Fetches also record the row’s ID or the S3 object key.