Test with real data

ProBusiness

When you test a workflow, you need a realistic input to walk through it. Rather than pasting sample JSON by hand, dagweave can pull a real record straight from a data source you connect: a row from a Postgres table, or an object from an S3 bucket. You pick one record, its fields pre-fill your workflow’s input parameters, and you run a single test through the graph to prove it behaves the way you expect.

This is part of the test-run flow, so it needs the connector and a plan that includes it. Free and Starter build and test with pasted samples instead.

Test runs are exempt from your execution quota, so you can test as often as you like without spending runs.

What it does

Open the test panel, expand Advanced: use a sample record, and switch to the From a connected source tab instead of pasting a sample. dagweave lists the sources you have registered, you pick one, and it shows a paged list of records from it. Select a record and press Use this item: dagweave fetches just that one record’s fields and drops them into the parameter table, matching field names to your workflow’s input parameters. From there you run the test and watch the per-node output, exactly as you would with a pasted sample.

Only sources bound to an online connector show up, since the connector is what actually reads the data. If a source’s connector is offline, its records cannot answer.

Two roles to set it up

Like the connector itself, this splits into a one-time operator task and the author’s day-to-day use.

The operator registers a source

A cluster operator does the setup, once per source:

Create a read credential as a Kubernetes Secret in the connector’s namespace. Point it at a least-privilege, read-only principal scoped to just the table or bucket you want authors to test against: a read-only Postgres user, or S3 keys with ListObjects/GetObject on one bucket. The Secret holds the credential; nothing else needs it.
Register the source in dagweave. You give it a name, pick the type (Postgres or S3), choose which connector can reach it, name the Secret that holds the read credential, and set the non-secret connection hints: for Postgres, the host, port, database, the table allowlist, and which columns to show in the preview list; for S3, the bucket, an optional key prefix, and the region.

dagweave stores only the reference to that Secret and the non-secret hints. The credential itself never leaves your cluster.

The author picks a record

A workflow author opens the test panel, switches to From a connected source, picks a source, browses the records, selects one, and runs the test. No cluster access and no knowledge of the Secret is required. They see only the previews the operator chose to expose and the one record they pick.

The security model

The point of this design is that your read credentials never leave your cluster.

Credentials stay in-cluster. The read credential lives as a Kubernetes Secret in the connector’s namespace. dagweave stores a reference to it (the Secret name), never the password or keys. A compromise of dagweave leaks the name of a source, not a way to connect to your database. This is the same posture dagweave already uses for artifacts and logs, where it records where the bytes are rather than copying them.
Reads are read-only and least-privilege. The connector only reads. Point the Secret at a read-only role scoped to the one table or bucket, and a source read can neither write nor reach beyond what you granted.
Postgres reads are injection-safe. The table and preview columns come from the allowlist the operator set at registration, not from the author. Queries are parameterized; an author never supplies raw SQL. A record is fetched by its key as a bound parameter.
S3 reads are scoped. Reads are confined to the configured bucket and optional key prefix, and a key outside that scope is rejected.
Only what you pick is kept. dagweave persists just the fields of the one record you select, and only as the test parameters for that run, handled the same as a sample you would have pasted. The browsed list and every record you did not pick are transient: they are never written to dagweave’s store.
Every read is audited. dagweave records who read which source and its connector, and when. The trail records the read, never the customer data it returned.

Source types

Two source types ship today:

Postgres. Pick a row from an allowlisted table. The preview list shows the columns the operator chose, and selecting a row fetches its full field set.
S3. Pick an object from the configured bucket and prefix. The object’s contents become the record you test with.

More source types may follow, one at a time, with the same read-only, credentials-stay-in-cluster model.